Nov 03 14:08:25.889073 ip-10-0-143-171 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 03 14:08:25.889250 ip-10-0-143-171 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 03 14:08:25.889347 ip-10-0-143-171 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 03 14:08:25.889849 ip-10-0-143-171 systemd[1]: Failed to start Kubernetes Kubelet. Nov 03 14:08:35.262749 ip-10-0-143-171 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found. Nov 03 14:08:35.262763 ip-10-0-143-171 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 6c77de0266f74bdbab9f9e527abc5356 -- Nov 03 14:10:22.358348 ip-10-0-143-171 systemd[1]: Starting Kubernetes Kubelet... Nov 03 14:10:22.777277 ip-10-0-143-171 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 14:10:22.777277 ip-10-0-143-171 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 14:10:22.777277 ip-10-0-143-171 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 14:10:22.777277 ip-10-0-143-171 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 14:10:22.781816 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781737 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 14:10:22.781816 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781816 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 14:10:22.781816 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781821 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 14:10:22.781907 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781824 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 14:10:22.781907 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781827 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 14:10:22.781907 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781830 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 14:10:22.781907 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781832 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 14:10:22.781907 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781835 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 14:10:22.781907 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781838 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 14:10:22.781907 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781840 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 14:10:22.781907 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781843 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 14:10:22.781907 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781846 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 14:10:22.781907 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781848 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 14:10:22.781907 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781851 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 14:10:22.781907 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781854 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 14:10:22.781907 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781857 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 14:10:22.781907 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781860 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 14:10:22.781907 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781863 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 14:10:22.781907 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781865 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 14:10:22.781907 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781868 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 14:10:22.781907 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781871 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 14:10:22.781907 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781874 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 14:10:22.781907 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781876 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 14:10:22.782393 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781879 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 14:10:22.782393 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781883 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 14:10:22.782393 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781885 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 14:10:22.782393 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781888 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 14:10:22.782393 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781891 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 14:10:22.782393 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781893 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 14:10:22.782393 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781896 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 14:10:22.782393 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781898 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 14:10:22.782393 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781901 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 14:10:22.782393 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781905 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 14:10:22.782393 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781907 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 14:10:22.782393 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781910 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 14:10:22.782393 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781914 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 14:10:22.782393 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781917 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 14:10:22.782393 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781920 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 14:10:22.782393 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781923 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 14:10:22.782393 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781925 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 14:10:22.782393 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781928 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 14:10:22.782393 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781931 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 14:10:22.782836 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781934 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 14:10:22.782836 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781936 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 14:10:22.782836 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781939 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 14:10:22.782836 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781943 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 14:10:22.782836 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781947 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 14:10:22.782836 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781949 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 14:10:22.782836 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781952 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 14:10:22.782836 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781955 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 14:10:22.782836 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781957 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 14:10:22.782836 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781960 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 14:10:22.782836 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781963 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 14:10:22.782836 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781965 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 14:10:22.782836 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781968 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 14:10:22.782836 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781970 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 14:10:22.782836 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781974 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 14:10:22.782836 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781977 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 14:10:22.782836 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781981 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 14:10:22.782836 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781985 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 14:10:22.782836 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781990 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 14:10:22.783368 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781993 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 14:10:22.783368 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781996 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 14:10:22.783368 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.781999 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 14:10:22.783368 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.782002 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 14:10:22.783368 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.782004 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 14:10:22.783368 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.782007 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 14:10:22.783368 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.782009 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 14:10:22.783368 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.782012 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 14:10:22.783368 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.782015 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 14:10:22.783368 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.782017 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 14:10:22.783368 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.782020 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 14:10:22.783368 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.782023 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 14:10:22.783368 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.782026 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 14:10:22.783368 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.782029 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 14:10:22.783368 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.782031 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 14:10:22.783368 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.782034 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 14:10:22.783368 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.782037 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 14:10:22.783368 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.782040 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 14:10:22.783368 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.782043 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 14:10:22.783368 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.782046 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 14:10:22.783836 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.782048 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 14:10:22.783836 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.782051 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 14:10:22.783836 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.782053 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 14:10:22.783836 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.782070 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 14:10:22.783836 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.782075 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 03 14:10:22.783836 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.782078 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 14:10:22.783836 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.782082 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 14:10:22.783836 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.782088 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 14:10:22.783836 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.782091 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 14:10:22.783836 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.782094 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 14:10:22.783836 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.782097 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 14:10:22.783836 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.782100 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 14:10:22.783836 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.782103 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 14:10:22.783836 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.782105 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 14:10:22.783836 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.782108 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 14:10:22.783836 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.782111 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 14:10:22.783836 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.782114 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 14:10:22.783836 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.782116 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 14:10:22.783836 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.782119 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 14:10:22.784301 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.782121 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 14:10:22.784301 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.782124 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 14:10:22.784301 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.782126 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 14:10:22.784301 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.782129 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 14:10:22.784301 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.782131 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 14:10:22.784301 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.782134 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 14:10:22.784301 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783791 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 14:10:22.784301 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783800 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 14:10:22.784301 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783804 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 14:10:22.784301 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783807 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 14:10:22.784301 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783811 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 14:10:22.784301 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783814 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 14:10:22.784301 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783816 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 14:10:22.784301 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783819 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 14:10:22.784301 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783821 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 14:10:22.784301 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783824 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 14:10:22.784301 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783827 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 14:10:22.784301 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783829 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 14:10:22.784301 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783832 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 14:10:22.784301 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783835 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 14:10:22.784778 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783838 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 14:10:22.784778 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783841 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 14:10:22.784778 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783844 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 14:10:22.784778 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783847 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 14:10:22.784778 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783850 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 14:10:22.784778 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783853 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 14:10:22.784778 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783855 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 14:10:22.784778 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783858 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 14:10:22.784778 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783860 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 14:10:22.784778 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783863 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 14:10:22.784778 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783865 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 14:10:22.784778 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783868 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 14:10:22.784778 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783870 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 14:10:22.784778 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783873 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 14:10:22.784778 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783876 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 14:10:22.784778 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783878 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 14:10:22.784778 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783881 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 14:10:22.784778 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783883 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 14:10:22.784778 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783886 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 14:10:22.784778 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783889 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 14:10:22.785261 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783891 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 14:10:22.785261 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783894 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 14:10:22.785261 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783897 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 03 14:10:22.785261 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783899 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 14:10:22.785261 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783902 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 14:10:22.785261 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783904 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 14:10:22.785261 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783907 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 14:10:22.785261 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783909 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 14:10:22.785261 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783912 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 14:10:22.785261 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783914 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 14:10:22.785261 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783917 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 14:10:22.785261 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783920 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 14:10:22.785261 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783923 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 14:10:22.785261 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783926 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 14:10:22.785261 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783928 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 14:10:22.785261 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783931 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 14:10:22.785261 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783933 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 14:10:22.785261 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783936 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 14:10:22.785261 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783939 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 14:10:22.785261 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783941 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 14:10:22.785752 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783944 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 14:10:22.785752 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783946 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 14:10:22.785752 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783949 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 14:10:22.785752 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783951 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 14:10:22.785752 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783954 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 14:10:22.785752 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783957 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 14:10:22.785752 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783959 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 14:10:22.785752 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783961 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 14:10:22.785752 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783964 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 14:10:22.785752 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783966 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 14:10:22.785752 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783969 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 14:10:22.785752 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783971 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 14:10:22.785752 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783974 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 14:10:22.785752 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783977 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 14:10:22.785752 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783981 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 14:10:22.785752 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783984 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 14:10:22.785752 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783987 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 14:10:22.785752 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783989 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 14:10:22.785752 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783992 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 14:10:22.786226 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783996 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 14:10:22.786226 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.783999 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 14:10:22.786226 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784002 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 14:10:22.786226 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784004 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 14:10:22.786226 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784007 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 14:10:22.786226 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784012 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 14:10:22.786226 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784016 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 14:10:22.786226 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784019 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 14:10:22.786226 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784021 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 14:10:22.786226 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784024 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 14:10:22.786226 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784026 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 14:10:22.786226 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784029 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 14:10:22.786226 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784032 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 14:10:22.786226 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784034 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 14:10:22.786226 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784037 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 14:10:22.786226 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784039 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 14:10:22.786226 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784042 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 14:10:22.786226 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784045 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 14:10:22.786226 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784047 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 14:10:22.786679 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784049 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 14:10:22.786679 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784052 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 14:10:22.786679 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784054 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 14:10:22.786679 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784071 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 14:10:22.786679 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784074 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 14:10:22.786679 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784077 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 14:10:22.786679 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784080 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 14:10:22.786679 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784084 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 14:10:22.786679 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784086 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 14:10:22.786679 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784089 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 14:10:22.786679 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784092 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 14:10:22.786679 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784095 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 14:10:22.786679 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784098 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 14:10:22.786679 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784100 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 14:10:22.786679 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784579 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 14:10:22.786679 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784584 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 14:10:22.786679 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784587 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 14:10:22.786679 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784590 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 14:10:22.786679 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784593 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 14:10:22.786679 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784596 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 14:10:22.787159 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784599 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 14:10:22.787159 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784601 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 14:10:22.787159 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784604 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 14:10:22.787159 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784608 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 14:10:22.787159 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784610 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 14:10:22.787159 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784613 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 14:10:22.787159 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784616 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 14:10:22.787159 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784618 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 14:10:22.787159 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784621 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 14:10:22.787159 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784623 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 14:10:22.787159 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784626 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 14:10:22.787159 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784628 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 14:10:22.787159 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784631 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 14:10:22.787159 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784634 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 14:10:22.787159 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784637 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 14:10:22.787159 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784640 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 14:10:22.787159 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784643 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 14:10:22.787159 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784646 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 14:10:22.787159 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784648 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 14:10:22.787159 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784652 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 14:10:22.787631 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784655 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 14:10:22.787631 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784658 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 14:10:22.787631 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784660 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 14:10:22.787631 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784663 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 14:10:22.787631 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784666 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 14:10:22.787631 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784668 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 14:10:22.787631 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784671 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 14:10:22.787631 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784673 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 14:10:22.787631 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784676 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 03 14:10:22.787631 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784678 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 14:10:22.787631 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784681 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 14:10:22.787631 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784684 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 14:10:22.787631 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784686 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 14:10:22.787631 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784689 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 14:10:22.787631 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784691 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 14:10:22.787631 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784694 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 14:10:22.787631 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784696 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 14:10:22.787631 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784699 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 14:10:22.787631 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784701 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 14:10:22.788084 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784706 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 14:10:22.788084 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784709 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 14:10:22.788084 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784712 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 14:10:22.788084 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784715 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 14:10:22.788084 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784718 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 14:10:22.788084 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784721 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 14:10:22.788084 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784723 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 14:10:22.788084 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784728 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 14:10:22.788084 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784731 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 14:10:22.788084 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784733 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 14:10:22.788084 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784736 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 14:10:22.788084 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784738 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 14:10:22.788084 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784741 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 14:10:22.788084 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784744 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 14:10:22.788084 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784746 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 14:10:22.788084 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784749 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 14:10:22.788084 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784751 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 14:10:22.788084 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784755 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 14:10:22.788084 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784757 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 14:10:22.789561 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784760 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 14:10:22.789561 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784762 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 14:10:22.789561 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784764 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 14:10:22.789561 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784767 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 14:10:22.789561 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784770 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 14:10:22.789561 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784774 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 14:10:22.789561 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784776 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 14:10:22.789561 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784780 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 14:10:22.789561 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784782 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 14:10:22.789561 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784785 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 14:10:22.789561 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784788 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 14:10:22.789561 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784790 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 14:10:22.789561 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784793 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 14:10:22.789561 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784796 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 14:10:22.789561 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784798 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 14:10:22.789561 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784801 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 14:10:22.789561 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784803 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 14:10:22.789561 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784806 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 14:10:22.789561 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784808 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 14:10:22.790014 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784811 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 14:10:22.790014 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784815 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 14:10:22.790014 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784818 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 14:10:22.790014 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784820 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 14:10:22.790014 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784823 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 14:10:22.790014 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784825 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 14:10:22.790014 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784828 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 14:10:22.790014 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784830 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 14:10:22.790014 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784833 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 14:10:22.790014 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784836 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 14:10:22.790014 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784839 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 14:10:22.790014 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784841 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 14:10:22.790014 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784844 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 14:10:22.790014 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784846 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 14:10:22.790014 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784849 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 14:10:22.790014 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784851 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 14:10:22.790014 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784854 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 14:10:22.790014 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784857 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 14:10:22.790014 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784859 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 14:10:22.790014 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784862 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 14:10:22.790509 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784865 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 14:10:22.790509 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784867 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 14:10:22.790509 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.784870 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 14:10:22.790509 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.785910 2577 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 14:10:22.790509 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.785920 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 14:10:22.790509 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.785928 2577 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 14:10:22.790509 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.785933 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 14:10:22.790509 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.785938 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 14:10:22.790509 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.785941 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 14:10:22.790509 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.785945 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 14:10:22.790509 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.785950 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 14:10:22.790509 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.785953 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 14:10:22.790509 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.785957 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 14:10:22.790509 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.785961 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 14:10:22.790509 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.785965 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 14:10:22.790509 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.785968 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 14:10:22.790509 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.785971 2577 flags.go:64] FLAG: --cgroup-root="" Nov 03 14:10:22.790509 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.785974 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 14:10:22.790509 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.785978 2577 flags.go:64] FLAG: --client-ca-file="" Nov 03 14:10:22.790509 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.785981 2577 flags.go:64] FLAG: --cloud-provider="external" Nov 03 14:10:22.790509 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.785984 2577 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 14:10:22.790509 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.785989 2577 flags.go:64] FLAG: --cluster-domain="" Nov 03 14:10:22.790509 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.785992 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 14:10:22.790509 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.785995 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 14:10:22.791107 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.785998 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 14:10:22.791107 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786002 2577 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 14:10:22.791107 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786006 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 14:10:22.791107 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786009 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 14:10:22.791107 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786012 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 14:10:22.791107 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786015 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 14:10:22.791107 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786019 2577 flags.go:64] FLAG: --contention-profiling="false" Nov 03 14:10:22.791107 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786022 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 14:10:22.791107 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786026 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 14:10:22.791107 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786029 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 14:10:22.791107 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786032 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 14:10:22.791107 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786036 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 14:10:22.791107 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786039 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 14:10:22.791107 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786042 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 14:10:22.791107 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786045 2577 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 14:10:22.791107 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786048 2577 flags.go:64] FLAG: --enable-server="true" Nov 03 14:10:22.791107 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786051 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 14:10:22.791107 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786069 2577 flags.go:64] FLAG: --event-burst="100" Nov 03 14:10:22.791107 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786074 2577 flags.go:64] FLAG: --event-qps="50" Nov 03 14:10:22.791107 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786079 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 14:10:22.791107 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786086 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 14:10:22.791107 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786090 2577 flags.go:64] FLAG: --eviction-hard="" Nov 03 14:10:22.791107 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786094 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 14:10:22.791107 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786097 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 14:10:22.791719 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786100 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 14:10:22.791719 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786104 2577 flags.go:64] FLAG: --eviction-soft="" Nov 03 14:10:22.791719 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786107 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 14:10:22.791719 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786110 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 14:10:22.791719 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786113 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 14:10:22.791719 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786116 2577 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 14:10:22.791719 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786120 2577 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 14:10:22.791719 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786123 2577 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 14:10:22.791719 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786126 2577 flags.go:64] FLAG: --feature-gates="" Nov 03 14:10:22.791719 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786130 2577 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 14:10:22.791719 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786133 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 14:10:22.791719 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786137 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 14:10:22.791719 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786140 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 14:10:22.791719 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786143 2577 flags.go:64] FLAG: --healthz-port="10248" Nov 03 14:10:22.791719 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786146 2577 flags.go:64] FLAG: --help="false" Nov 03 14:10:22.791719 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786149 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-143-171.ec2.internal" Nov 03 14:10:22.791719 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786152 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 14:10:22.791719 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786155 2577 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 14:10:22.791719 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786158 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 14:10:22.791719 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786161 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 14:10:22.791719 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786165 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 14:10:22.791719 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786168 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 14:10:22.791719 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786171 2577 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 14:10:22.792293 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786174 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 14:10:22.792293 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786176 2577 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 14:10:22.792293 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786179 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 14:10:22.792293 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786183 2577 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 14:10:22.792293 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786185 2577 flags.go:64] FLAG: --kube-reserved="" Nov 03 14:10:22.792293 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786190 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 14:10:22.792293 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786193 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 14:10:22.792293 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786196 2577 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 14:10:22.792293 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786199 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 14:10:22.792293 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786202 2577 flags.go:64] FLAG: --lock-file="" Nov 03 14:10:22.792293 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786205 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 14:10:22.792293 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786208 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 14:10:22.792293 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786211 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 14:10:22.792293 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786217 2577 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 14:10:22.792293 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786220 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 14:10:22.792293 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786223 2577 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 14:10:22.792293 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786226 2577 flags.go:64] FLAG: --logging-format="text" Nov 03 14:10:22.792293 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786230 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 14:10:22.792293 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786233 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 14:10:22.792293 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786237 2577 flags.go:64] FLAG: --manifest-url="" Nov 03 14:10:22.792293 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786240 2577 flags.go:64] FLAG: --manifest-url-header="" Nov 03 14:10:22.792293 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786244 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 14:10:22.792293 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786248 2577 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 14:10:22.792293 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786252 2577 flags.go:64] FLAG: --max-pods="110" Nov 03 14:10:22.792293 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786255 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 14:10:22.792896 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786258 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 14:10:22.792896 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786261 2577 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 14:10:22.792896 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786264 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 14:10:22.792896 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786267 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 14:10:22.792896 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786270 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 14:10:22.792896 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786273 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 14:10:22.792896 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786280 2577 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 14:10:22.792896 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786284 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 14:10:22.792896 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786287 2577 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 14:10:22.792896 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786290 2577 flags.go:64] FLAG: --pod-cidr="" Nov 03 14:10:22.792896 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786293 2577 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 14:10:22.792896 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786296 2577 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 14:10:22.792896 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786300 2577 flags.go:64] FLAG: --pods-per-core="0" Nov 03 14:10:22.792896 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786303 2577 flags.go:64] FLAG: --port="10250" Nov 03 14:10:22.792896 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786306 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 14:10:22.792896 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786309 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c10c1b9db03ccfbc" Nov 03 14:10:22.792896 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786312 2577 flags.go:64] FLAG: --qos-reserved="" Nov 03 14:10:22.792896 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786315 2577 flags.go:64] FLAG: --read-only-port="10255" Nov 03 14:10:22.792896 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786318 2577 flags.go:64] FLAG: --register-node="true" Nov 03 14:10:22.792896 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786321 2577 flags.go:64] FLAG: --register-with-taints="" Nov 03 14:10:22.792896 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786325 2577 flags.go:64] FLAG: --registry-burst="10" Nov 03 14:10:22.792896 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786328 2577 flags.go:64] FLAG: --registry-qps="5" Nov 03 14:10:22.792896 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786331 2577 flags.go:64] FLAG: --reserved-cpus="" Nov 03 14:10:22.792896 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786334 2577 flags.go:64] FLAG: --reserved-memory="" Nov 03 14:10:22.792896 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786339 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 14:10:22.792896 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786342 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 14:10:22.793649 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786345 2577 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 14:10:22.793649 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786348 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 14:10:22.793649 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786351 2577 flags.go:64] FLAG: --runonce="false" Nov 03 14:10:22.793649 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786354 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 14:10:22.793649 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786357 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 14:10:22.793649 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786360 2577 flags.go:64] FLAG: --seccomp-default="false" Nov 03 14:10:22.793649 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786363 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 14:10:22.793649 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786366 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 14:10:22.793649 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786369 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 14:10:22.793649 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786372 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 14:10:22.793649 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786375 2577 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 14:10:22.793649 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786378 2577 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 14:10:22.793649 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786381 2577 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 14:10:22.793649 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786384 2577 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 14:10:22.793649 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786387 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 14:10:22.793649 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786390 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 14:10:22.793649 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786393 2577 flags.go:64] FLAG: --system-cgroups="" Nov 03 14:10:22.793649 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786395 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 14:10:22.793649 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786402 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 14:10:22.793649 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786406 2577 flags.go:64] FLAG: --tls-cert-file="" Nov 03 14:10:22.793649 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786409 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 14:10:22.793649 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786414 2577 flags.go:64] FLAG: --tls-min-version="" Nov 03 14:10:22.793649 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786416 2577 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 14:10:22.793649 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786419 2577 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 14:10:22.793649 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786422 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 14:10:22.794491 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786425 2577 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 14:10:22.794491 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786428 2577 flags.go:64] FLAG: --v="2" Nov 03 14:10:22.794491 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786433 2577 flags.go:64] FLAG: --version="false" Nov 03 14:10:22.794491 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786437 2577 flags.go:64] FLAG: --vmodule="" Nov 03 14:10:22.794491 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786441 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 14:10:22.794491 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786444 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 14:10:22.794491 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786580 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 14:10:22.794491 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786584 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 14:10:22.794491 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786587 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 14:10:22.794491 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786590 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 14:10:22.794491 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786592 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 14:10:22.794491 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786595 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 14:10:22.794491 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786599 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 14:10:22.794491 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786604 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 14:10:22.794491 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786607 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 14:10:22.794491 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786610 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 14:10:22.794491 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786612 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 14:10:22.794491 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786615 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 14:10:22.794491 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786617 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 14:10:22.794491 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786620 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 14:10:22.794970 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786623 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 14:10:22.794970 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786625 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 14:10:22.794970 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786628 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 14:10:22.794970 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786631 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 14:10:22.794970 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786633 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 14:10:22.794970 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786637 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 14:10:22.794970 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786642 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 14:10:22.794970 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786646 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 14:10:22.794970 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786650 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 14:10:22.794970 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786653 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 14:10:22.794970 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786656 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 14:10:22.794970 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786659 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 14:10:22.794970 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786661 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 14:10:22.794970 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786664 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 14:10:22.794970 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786666 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 14:10:22.794970 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786669 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 14:10:22.794970 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786672 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 14:10:22.794970 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786674 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 14:10:22.794970 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786677 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 14:10:22.795444 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786680 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 14:10:22.795444 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786683 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 14:10:22.795444 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786685 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 14:10:22.795444 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786688 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 14:10:22.795444 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786692 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 14:10:22.795444 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786694 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 14:10:22.795444 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786697 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 14:10:22.795444 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786699 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 14:10:22.795444 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786702 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 14:10:22.795444 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786704 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 14:10:22.795444 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786707 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 14:10:22.795444 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786709 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 14:10:22.795444 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786712 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 03 14:10:22.795444 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786714 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 14:10:22.795444 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786717 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 14:10:22.795444 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786720 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 14:10:22.795444 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786722 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 14:10:22.795444 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786725 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 14:10:22.795444 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786728 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 14:10:22.795444 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786735 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 14:10:22.795942 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786738 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 14:10:22.795942 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786740 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 14:10:22.795942 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786743 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 14:10:22.795942 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786746 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 14:10:22.795942 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786749 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 14:10:22.795942 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786751 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 14:10:22.795942 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786754 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 14:10:22.795942 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786757 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 14:10:22.795942 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786759 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 14:10:22.795942 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786762 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 14:10:22.795942 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786765 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 14:10:22.795942 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786767 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 14:10:22.795942 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786770 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 14:10:22.795942 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786773 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 14:10:22.795942 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786776 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 14:10:22.795942 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786778 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 14:10:22.795942 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786781 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 14:10:22.795942 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786783 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 14:10:22.795942 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786786 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 14:10:22.796405 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786788 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 14:10:22.796405 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786791 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 14:10:22.796405 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786794 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 14:10:22.796405 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786796 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 14:10:22.796405 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786799 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 14:10:22.796405 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786801 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 14:10:22.796405 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786804 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 14:10:22.796405 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786806 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 14:10:22.796405 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786809 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 14:10:22.796405 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786811 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 14:10:22.796405 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786814 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 14:10:22.796405 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786816 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 14:10:22.796405 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786819 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 14:10:22.796405 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786822 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 14:10:22.796405 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786825 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 14:10:22.796405 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786827 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 14:10:22.796405 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786830 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 14:10:22.796405 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786832 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 14:10:22.796405 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786834 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 14:10:22.796405 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786837 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 14:10:22.796876 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786840 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 14:10:22.796876 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786842 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 14:10:22.796876 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786844 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 14:10:22.796876 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786847 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 14:10:22.796876 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786849 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 14:10:22.796876 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786852 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 14:10:22.796876 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786854 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 14:10:22.796876 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786857 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 14:10:22.796876 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786860 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 14:10:22.796876 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786862 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 14:10:22.796876 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786865 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 14:10:22.796876 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786868 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 14:10:22.796876 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786870 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 14:10:22.796876 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786873 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 14:10:22.796876 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.786899 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 14:10:22.796876 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786941 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 14:10:22.797275 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786946 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 14:10:22.797275 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786950 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 14:10:22.797275 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786953 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 14:10:22.797275 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786956 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 14:10:22.797275 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786959 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 14:10:22.797275 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786961 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 14:10:22.797275 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786964 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 14:10:22.797275 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786966 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 14:10:22.797275 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786970 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 14:10:22.797275 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786973 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 14:10:22.797275 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786976 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 14:10:22.797275 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786978 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 14:10:22.797275 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786981 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 14:10:22.797275 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786983 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 14:10:22.797275 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786986 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 14:10:22.797275 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786988 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 14:10:22.797275 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786991 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 14:10:22.797275 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786993 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 14:10:22.797275 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786996 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 14:10:22.797731 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.786998 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 14:10:22.797731 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787002 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 14:10:22.797731 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787004 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 14:10:22.797731 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787007 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 14:10:22.797731 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787009 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 14:10:22.797731 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787012 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 03 14:10:22.797731 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787015 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 14:10:22.797731 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787018 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 14:10:22.797731 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787020 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 14:10:22.797731 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787023 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 14:10:22.797731 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787026 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 14:10:22.797731 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787028 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 14:10:22.797731 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787031 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 14:10:22.797731 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787033 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 14:10:22.797731 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787036 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 14:10:22.797731 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787038 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 14:10:22.797731 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787041 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 14:10:22.797731 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787044 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 14:10:22.797731 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787046 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 14:10:22.797731 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787049 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 14:10:22.798220 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787051 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 14:10:22.798220 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787055 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 14:10:22.798220 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787072 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 14:10:22.798220 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787075 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 14:10:22.798220 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787078 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 14:10:22.798220 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787081 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 14:10:22.798220 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787084 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 14:10:22.798220 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787087 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 14:10:22.798220 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787089 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 14:10:22.798220 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787092 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 14:10:22.798220 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787094 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 14:10:22.798220 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787097 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 14:10:22.798220 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787099 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 14:10:22.798220 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787102 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 14:10:22.798220 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787105 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 14:10:22.798220 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787107 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 14:10:22.798220 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787110 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 14:10:22.798220 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787112 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 14:10:22.798220 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787115 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 14:10:22.798672 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787118 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 14:10:22.798672 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787121 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 14:10:22.798672 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787123 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 14:10:22.798672 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787127 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 14:10:22.798672 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787130 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 14:10:22.798672 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787133 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 14:10:22.798672 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787135 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 14:10:22.798672 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787138 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 14:10:22.798672 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787140 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 14:10:22.798672 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787143 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 14:10:22.798672 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787146 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 14:10:22.798672 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787148 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 14:10:22.798672 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787151 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 14:10:22.798672 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787166 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 14:10:22.798672 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787173 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 14:10:22.798672 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787178 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 14:10:22.798672 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787182 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 14:10:22.798672 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787184 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 14:10:22.798672 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787187 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 14:10:22.798672 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787189 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 14:10:22.799190 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787192 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 14:10:22.799190 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787195 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 14:10:22.799190 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787197 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 14:10:22.799190 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787200 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 14:10:22.799190 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787203 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 14:10:22.799190 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787205 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 14:10:22.799190 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787208 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 14:10:22.799190 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787210 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 14:10:22.799190 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787213 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 14:10:22.799190 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787215 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 14:10:22.799190 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787218 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 14:10:22.799190 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787221 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 14:10:22.799190 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787225 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 14:10:22.799190 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787229 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 14:10:22.799190 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787232 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 14:10:22.799190 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787237 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 14:10:22.799190 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787240 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 14:10:22.799190 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787243 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 14:10:22.799190 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787245 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 14:10:22.799630 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787248 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 14:10:22.799630 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787251 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 14:10:22.799630 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787253 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 14:10:22.799630 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787256 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 14:10:22.799630 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787259 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 14:10:22.799630 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787261 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 14:10:22.799630 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787264 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 14:10:22.799630 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.787268 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 14:10:22.799630 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.795590 2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 14:10:22.799630 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.795612 2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 14:10:22.799630 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.796396 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 14:10:22.799630 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.796404 2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 14:10:22.799630 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796466 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 14:10:22.799630 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796471 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 14:10:22.799630 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796474 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 14:10:22.799630 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796477 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 14:10:22.799630 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796480 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 14:10:22.799630 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796483 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 14:10:22.799630 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796486 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 14:10:22.799630 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796489 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 14:10:22.800105 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796491 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 14:10:22.800105 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796494 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 14:10:22.800105 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796496 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 14:10:22.800105 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796501 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 14:10:22.800105 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796504 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 14:10:22.800105 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796507 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 14:10:22.800105 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796510 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 14:10:22.800105 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796513 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 14:10:22.800105 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796515 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 14:10:22.800105 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796519 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 14:10:22.800105 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796521 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 14:10:22.800105 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796524 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 14:10:22.800105 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796527 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 14:10:22.800105 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796536 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 14:10:22.800105 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796538 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 14:10:22.800105 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796541 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 14:10:22.800105 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796544 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 14:10:22.800105 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796546 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 14:10:22.800105 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796549 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 14:10:22.800542 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796552 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 14:10:22.800542 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796556 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 14:10:22.800542 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796559 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 14:10:22.800542 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796561 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 14:10:22.800542 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796564 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 14:10:22.800542 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796567 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 14:10:22.800542 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796569 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 14:10:22.800542 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796572 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 14:10:22.800542 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796574 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 14:10:22.800542 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796577 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 14:10:22.800542 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796580 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 14:10:22.800542 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796583 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 14:10:22.800542 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796586 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 14:10:22.800542 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796588 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 14:10:22.800542 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796591 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 14:10:22.800542 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796594 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 14:10:22.800542 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796596 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 14:10:22.800542 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796599 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 14:10:22.800542 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796601 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 14:10:22.800542 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796604 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 14:10:22.801006 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796606 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 14:10:22.801006 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796609 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 14:10:22.801006 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796611 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 14:10:22.801006 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796614 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 14:10:22.801006 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796616 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 14:10:22.801006 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796619 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 14:10:22.801006 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796622 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 14:10:22.801006 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796625 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 14:10:22.801006 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796627 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 03 14:10:22.801006 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796630 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 14:10:22.801006 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796633 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 14:10:22.801006 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796636 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 14:10:22.801006 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796638 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 14:10:22.801006 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796641 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 14:10:22.801006 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796644 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 14:10:22.801006 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796647 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 14:10:22.801006 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796650 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 14:10:22.801006 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796652 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 14:10:22.801006 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796655 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 14:10:22.801542 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796658 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 14:10:22.801542 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796660 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 14:10:22.801542 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796663 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 14:10:22.801542 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796666 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 14:10:22.801542 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796668 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 14:10:22.801542 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796671 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 14:10:22.801542 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796673 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 14:10:22.801542 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796676 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 14:10:22.801542 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796679 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 14:10:22.801542 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796681 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 14:10:22.801542 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796684 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 14:10:22.801542 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796687 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 14:10:22.801542 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796689 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 14:10:22.801542 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796692 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 14:10:22.801542 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796695 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 14:10:22.801542 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796698 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 14:10:22.801542 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796700 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 14:10:22.801542 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796703 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 14:10:22.801542 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796705 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 14:10:22.801982 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796707 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 14:10:22.801982 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796711 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 14:10:22.801982 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796713 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 14:10:22.801982 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796716 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 14:10:22.801982 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796719 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 14:10:22.801982 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796721 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 14:10:22.801982 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796724 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 14:10:22.801982 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796726 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 14:10:22.801982 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796729 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 14:10:22.801982 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796734 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 14:10:22.801982 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796738 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 14:10:22.801982 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796742 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 14:10:22.801982 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796745 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 14:10:22.801982 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796748 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 14:10:22.801982 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796751 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 14:10:22.801982 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796754 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 14:10:22.801982 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796756 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 14:10:22.801982 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796759 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 14:10:22.801982 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796761 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 14:10:22.802445 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796764 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 14:10:22.802445 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796767 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 14:10:22.802445 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.796794 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 14:10:22.802445 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796931 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 14:10:22.802445 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796936 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 14:10:22.802445 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796940 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 14:10:22.802445 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796943 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 14:10:22.802445 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796946 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 14:10:22.802445 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796951 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 14:10:22.802445 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796954 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 14:10:22.802445 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796957 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 14:10:22.802445 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796960 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 14:10:22.802445 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796964 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 14:10:22.802445 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796966 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 14:10:22.802445 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796969 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 14:10:22.802445 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796972 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 14:10:22.802817 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796975 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 14:10:22.802817 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796978 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 14:10:22.802817 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796980 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 14:10:22.802817 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796983 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 14:10:22.802817 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796986 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 14:10:22.802817 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796989 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 14:10:22.802817 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796991 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 14:10:22.802817 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796994 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 14:10:22.802817 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.796997 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 14:10:22.802817 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797000 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 14:10:22.802817 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797002 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 14:10:22.802817 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797005 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 14:10:22.802817 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797008 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 14:10:22.802817 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797010 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 14:10:22.802817 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797013 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 14:10:22.802817 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797016 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 14:10:22.802817 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797018 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 14:10:22.802817 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797021 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 14:10:22.802817 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797023 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 14:10:22.802817 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797026 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 14:10:22.803434 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797029 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 14:10:22.803434 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797031 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 14:10:22.803434 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797035 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 14:10:22.803434 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797038 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 14:10:22.803434 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797041 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 14:10:22.803434 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797045 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 14:10:22.803434 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797047 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 14:10:22.803434 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797050 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 14:10:22.803434 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797053 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 14:10:22.803434 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797055 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 14:10:22.803434 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797072 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 14:10:22.803434 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797075 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 14:10:22.803434 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797079 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 14:10:22.803434 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797083 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 14:10:22.803434 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797086 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 14:10:22.803434 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797088 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 14:10:22.803434 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797091 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 14:10:22.803434 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797094 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 14:10:22.803434 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797096 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 14:10:22.803872 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797099 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 14:10:22.803872 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797101 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 14:10:22.803872 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797104 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 14:10:22.803872 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797107 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 14:10:22.803872 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797109 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 14:10:22.803872 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797112 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 14:10:22.803872 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797114 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 14:10:22.803872 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797117 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 14:10:22.803872 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797120 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 14:10:22.803872 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797122 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 14:10:22.803872 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797125 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 14:10:22.803872 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797127 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 14:10:22.803872 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797130 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 14:10:22.803872 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797135 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 14:10:22.803872 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797138 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 03 14:10:22.803872 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797141 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 14:10:22.803872 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797143 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 14:10:22.803872 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797146 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 14:10:22.803872 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797149 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 14:10:22.804328 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797152 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 14:10:22.804328 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797155 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 14:10:22.804328 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797157 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 14:10:22.804328 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797160 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 14:10:22.804328 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797162 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 14:10:22.804328 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797165 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 14:10:22.804328 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797168 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 14:10:22.804328 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797171 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 14:10:22.804328 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797173 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 14:10:22.804328 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797176 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 14:10:22.804328 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797178 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 14:10:22.804328 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797181 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 14:10:22.804328 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797183 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 14:10:22.804328 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797186 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 14:10:22.804328 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797189 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 14:10:22.804328 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797191 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 14:10:22.804328 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797194 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 14:10:22.804328 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797197 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 14:10:22.804328 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797199 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 14:10:22.804767 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797202 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 14:10:22.804767 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797204 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 14:10:22.804767 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797207 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 14:10:22.804767 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797209 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 14:10:22.804767 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797212 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 14:10:22.804767 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797214 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 14:10:22.804767 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797217 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 14:10:22.804767 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797223 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 14:10:22.804767 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797225 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 14:10:22.804767 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797228 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 14:10:22.804767 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797231 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 14:10:22.804767 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797233 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 14:10:22.804767 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797236 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 14:10:22.804767 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797238 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 14:10:22.804767 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797241 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 14:10:22.804767 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797244 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 14:10:22.805151 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.797268 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 14:10:22.805151 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797299 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 14:10:22.805151 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797304 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 14:10:22.805151 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797307 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 14:10:22.805151 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797310 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 14:10:22.805151 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797314 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 14:10:22.805151 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797317 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 14:10:22.805151 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797320 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 14:10:22.805151 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797323 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 14:10:22.805151 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797325 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 14:10:22.805151 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797328 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 14:10:22.805151 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797330 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 14:10:22.805151 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797333 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 14:10:22.805151 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797335 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 14:10:22.805151 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797338 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 14:10:22.805151 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797341 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 14:10:22.805529 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797344 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 14:10:22.805529 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797346 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 14:10:22.805529 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797349 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 14:10:22.805529 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797351 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 14:10:22.805529 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797354 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 03 14:10:22.805529 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797356 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 14:10:22.805529 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797359 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 14:10:22.805529 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797362 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 14:10:22.805529 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797365 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 14:10:22.805529 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797367 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 14:10:22.805529 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797370 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 14:10:22.805529 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797372 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 14:10:22.805529 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797375 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 14:10:22.805529 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797377 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 14:10:22.805529 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797380 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 14:10:22.805529 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797383 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 14:10:22.805529 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797386 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 14:10:22.805529 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797388 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 14:10:22.805529 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797391 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 14:10:22.805529 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797393 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 14:10:22.806015 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797396 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 14:10:22.806015 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797399 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 14:10:22.806015 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797401 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 14:10:22.806015 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797404 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 14:10:22.806015 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797406 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 14:10:22.806015 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797409 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 14:10:22.806015 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797411 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 14:10:22.806015 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797414 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 14:10:22.806015 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797417 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 14:10:22.806015 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797419 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 14:10:22.806015 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797422 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 14:10:22.806015 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797424 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 14:10:22.806015 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797427 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 14:10:22.806015 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797429 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 14:10:22.806015 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797432 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 14:10:22.806015 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797434 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 14:10:22.806015 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797437 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 14:10:22.806015 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797439 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 14:10:22.806015 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797441 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 14:10:22.806469 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797445 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 14:10:22.806469 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797447 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 14:10:22.806469 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797450 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 14:10:22.806469 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797453 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 14:10:22.806469 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797456 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 14:10:22.806469 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797458 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 14:10:22.806469 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797461 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 14:10:22.806469 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797464 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 14:10:22.806469 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797466 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 14:10:22.806469 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797469 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 14:10:22.806469 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797472 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 14:10:22.806469 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797480 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 14:10:22.806469 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797484 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 14:10:22.806469 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797486 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 14:10:22.806469 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797489 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 14:10:22.806469 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797491 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 14:10:22.806469 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797494 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 14:10:22.806469 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797497 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 14:10:22.806469 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797500 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 14:10:22.806469 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797503 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 14:10:22.806939 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797505 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 14:10:22.806939 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797508 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 14:10:22.806939 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797510 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 14:10:22.806939 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797513 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 14:10:22.806939 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797515 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 14:10:22.806939 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797518 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 14:10:22.806939 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797520 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 14:10:22.806939 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797523 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 14:10:22.806939 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797525 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 14:10:22.806939 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797528 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 14:10:22.806939 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797530 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 14:10:22.806939 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797533 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 14:10:22.806939 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797537 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 14:10:22.806939 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797539 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 14:10:22.806939 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797542 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 14:10:22.806939 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797544 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 14:10:22.806939 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797547 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 14:10:22.806939 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797549 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 14:10:22.806939 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797552 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 14:10:22.806939 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797555 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 14:10:22.807420 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797559 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 14:10:22.807420 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797563 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 14:10:22.807420 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797567 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 14:10:22.807420 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797570 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 14:10:22.807420 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797573 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 14:10:22.807420 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797575 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 14:10:22.807420 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797578 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 14:10:22.807420 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797581 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 14:10:22.807420 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797583 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 14:10:22.807420 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797586 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 14:10:22.807420 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797589 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 14:10:22.807420 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:22.797593 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 14:10:22.807420 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.797653 2577 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 14:10:22.807420 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.800183 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 14:10:22.807420 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.801281 2577 server.go:1014] "Starting client certificate rotation" Nov 03 14:10:22.807420 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.801373 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 14:10:22.807420 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.801407 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 14:10:22.807846 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.803482 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 14:10:22.808630 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.808613 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 14:10:22.818229 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.818214 2577 log.go:25] "Validated CRI v1 runtime API" Nov 03 14:10:22.821421 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.821406 2577 log.go:25] "Validated CRI v1 image API" Nov 03 14:10:22.822215 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.822191 2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 14:10:22.829011 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.828988 2577 fs.go:135] Filesystem UUIDs: map[35141d3c-382c-4f7d-ac36-de190683c446:/dev/nvme0n1p3 37e8a46c-f295-4990-8032-e820db68c14f:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 03 14:10:22.829085 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.829011 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 14:10:22.835286 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.835174 2577 manager.go:217] Machine: {Timestamp:2026-11-03 14:10:22.83371697 +0000 UTC m=+0.380618357 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100171 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec263f13a03ab563fedaf683c8594dba SystemUUID:ec263f13-a03a-b563-feda-f683c8594dba BootID:6c77de02-66f7-4bdb-ab9f-9e527abc5356 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d8:63:14:e5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d8:63:14:e5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:86:93:f7:8e:47:3b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 14:10:22.836097 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.836086 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 14:10:22.836207 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.836196 2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 14:10:22.840716 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.840684 2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 14:10:22.840890 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.840716 2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-171.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 14:10:22.840948 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.840892 2577 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 14:10:22.840948 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.840901 2577 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 14:10:22.840948 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.840936 2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 14:10:22.841021 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.840956 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 14:10:22.842273 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.842262 2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 14:10:22.847049 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.847037 2577 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 14:10:22.847183 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.847173 2577 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 14:10:22.848022 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.848012 2577 kubelet.go:493] "Attempting to sync node with API server" Nov 03 14:10:22.848067 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.848025 2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 14:10:22.848067 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.848036 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 14:10:22.848067 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.848045 2577 kubelet.go:405] "Adding apiserver pod source" Nov 03 14:10:22.848067 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.848053 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 14:10:22.848404 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.848392 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 14:10:22.850847 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.850827 2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 14:10:22.857482 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.857465 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 14:10:22.858417 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.858405 2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 14:10:22.858460 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.858436 2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 14:10:22.859789 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.859780 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 14:10:22.859822 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.859794 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 14:10:22.859822 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.859800 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 14:10:22.859822 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.859805 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 14:10:22.859822 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.859811 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 14:10:22.859822 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.859817 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 14:10:22.860555 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.860547 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 14:10:22.860585 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.860555 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 14:10:22.860585 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.860563 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 14:10:22.860585 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.860576 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 14:10:22.860585 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.860584 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 14:10:22.860761 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.860593 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 14:10:22.862301 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.862287 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 14:10:22.862341 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.862314 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 14:10:22.866355 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.866340 2577 server.go:1263] "Started kubelet" Nov 03 14:10:22.866457 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.866407 2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 14:10:22.866513 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.866443 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 14:10:22.866561 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.866516 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 14:10:22.867091 ip-10-0-143-171 systemd[1]: Started Kubernetes Kubelet. Nov 03 14:10:22.868012 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.867990 2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 14:10:22.872507 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:22.871405 2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 14:10:22.873235 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.873224 2577 server.go:317] "Adding debug handlers to kubelet server" Nov 03 14:10:22.878431 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.878410 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 14:10:22.879038 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.879025 2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 14:10:22.879285 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:22.879262 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-171.ec2.internal\" not found" Nov 03 14:10:22.879772 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.879757 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 14:10:22.879772 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.879759 2577 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 14:10:22.879892 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.879780 2577 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 14:10:22.879978 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.879962 2577 reconstruct.go:97] "Volume reconstruction finished" Nov 03 14:10:22.879978 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.879977 2577 reconciler.go:29] "Reconciler: start to sync state" Nov 03 14:10:22.881461 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.881442 2577 factory.go:55] Registering systemd factory Nov 03 14:10:22.881461 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.881462 2577 factory.go:223] Registration of the systemd container factory successfully Nov 03 14:10:22.881755 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.881737 2577 factory.go:153] Registering CRI-O factory Nov 03 14:10:22.881755 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.881751 2577 factory.go:223] Registration of the crio container factory successfully Nov 03 14:10:22.881875 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.881799 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 14:10:22.881875 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.881823 2577 factory.go:103] Registering Raw factory Nov 03 14:10:22.881875 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.881837 2577 manager.go:1199] Started watching for new ooms in manager Nov 03 14:10:22.882784 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.882771 2577 manager.go:319] Starting recovery of all containers Nov 03 14:10:22.895106 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.895089 2577 manager.go:324] Recovery completed Nov 03 14:10:22.895621 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.895600 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-171.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 14:10:22.895700 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:22.895602 2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-171.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 14:10:22.897025 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:22.895787 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-171.ec2.internal.18e48e4f4e3a6172 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-171.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-171.ec2.internal,},FirstTimestamp:2026-11-03 14:10:22.86630949 +0000 UTC m=+0.413210877,LastTimestamp:2026-11-03 14:10:22.86630949 +0000 UTC m=+0.413210877,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-171.ec2.internal,}" Nov 03 14:10:22.900807 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.900792 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 14:10:22.902503 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.902487 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-171.ec2.internal" event="NodeHasSufficientMemory" Nov 03 14:10:22.902570 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.902514 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-171.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 14:10:22.902570 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.902524 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-171.ec2.internal" event="NodeHasSufficientPID" Nov 03 14:10:22.903086 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.903073 2577 cpu_manager.go:226] "Starting" policy="none" Nov 03 14:10:22.903086 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.903087 2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 14:10:22.903175 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.903102 2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 14:10:22.903175 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.903076 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 14:10:22.903835 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.903816 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-171.ec2.internal" event="NodeHasSufficientMemory" Nov 03 14:10:22.903912 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.903845 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-171.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 14:10:22.903912 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.903860 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-171.ec2.internal" event="NodeHasSufficientPID" Nov 03 14:10:22.905274 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.905255 2577 policy_none.go:50] "Start" Nov 03 14:10:22.905274 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.905270 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 14:10:22.905419 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.905279 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 14:10:22.906464 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.906446 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 14:10:22.907225 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.907211 2577 policy_none.go:44] "Start" Nov 03 14:10:22.913700 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:22.913675 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-171.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 14:10:22.913700 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:22.913675 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 14:10:22.913852 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:22.913675 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 03 14:10:22.933046 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:22.932963 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-171.ec2.internal.18e48e4f5062a0c8 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-171.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-171.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-171.ec2.internal,},FirstTimestamp:2026-11-03 14:10:22.902501576 +0000 UTC m=+0.449402963,LastTimestamp:2026-11-03 14:10:22.902501576 +0000 UTC m=+0.449402963,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-171.ec2.internal,}" Nov 03 14:10:22.960863 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.943632 2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 14:10:22.960863 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.943786 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r85z6" Nov 03 14:10:22.960863 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.946212 2577 manager.go:341] "Starting Device Plugin manager" Nov 03 14:10:22.960863 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:22.946236 2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 14:10:22.960863 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.946245 2577 server.go:87] "Starting device plugin registration server" Nov 03 14:10:22.960863 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.946462 2577 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 14:10:22.960863 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.946470 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 14:10:22.960863 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.946616 2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 14:10:22.960863 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.946681 2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 14:10:22.960863 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.946686 2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 14:10:22.960863 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:22.947133 2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 14:10:22.960863 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:22.947184 2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-171.ec2.internal\" not found" Nov 03 14:10:22.960863 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:22.947494 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-171.ec2.internal.18e48e4f5062e435 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-171.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-171.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-171.ec2.internal,},FirstTimestamp:2026-11-03 14:10:22.902518837 +0000 UTC m=+0.449420224,LastTimestamp:2026-11-03 14:10:22.902518837 +0000 UTC m=+0.449420224,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-171.ec2.internal,}" Nov 03 14:10:22.960863 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.954108 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r85z6" Nov 03 14:10:22.960863 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.954689 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 14:10:22.960863 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.955829 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 14:10:22.960863 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.955855 2577 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 14:10:22.960863 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:22.955878 2577 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 14:10:22.960863 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:22.955918 2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 14:10:22.963335 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:22.961309 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-171.ec2.internal.18e48e4f506306a5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-171.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-171.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-171.ec2.internal,},FirstTimestamp:2026-11-03 14:10:22.902527653 +0000 UTC m=+0.449429041,LastTimestamp:2026-11-03 14:10:22.902527653 +0000 UTC m=+0.449429041,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-171.ec2.internal,}" Nov 03 14:10:23.009666 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:23.009645 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 14:10:23.046659 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:23.046601 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 14:10:23.048159 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:23.048144 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-171.ec2.internal" event="NodeHasSufficientMemory" Nov 03 14:10:23.048229 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:23.048172 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-171.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 14:10:23.048229 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:23.048182 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-171.ec2.internal" event="NodeHasSufficientPID" Nov 03 14:10:23.048229 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:23.048206 2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-171.ec2.internal" Nov 03 14:10:23.056309 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:23.056289 2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-171.ec2.internal"] Nov 03 14:10:23.056361 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:23.056350 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 14:10:23.057138 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:23.057124 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-171.ec2.internal" event="NodeHasSufficientMemory" Nov 03 14:10:23.057194 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:23.057154 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-171.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 14:10:23.057194 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:23.057168 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-171.ec2.internal" event="NodeHasSufficientPID" Nov 03 14:10:23.059364 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:23.059353 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 14:10:23.059470 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:23.059455 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal" Nov 03 14:10:23.059507 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:23.059491 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 14:10:23.060054 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:23.060042 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-171.ec2.internal" event="NodeHasSufficientMemory" Nov 03 14:10:23.060133 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:23.060082 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-171.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 14:10:23.060133 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:23.060094 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-171.ec2.internal" event="NodeHasSufficientPID" Nov 03 14:10:23.060205 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:23.060153 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-171.ec2.internal" event="NodeHasSufficientMemory" Nov 03 14:10:23.060205 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:23.060173 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-171.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 14:10:23.060205 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:23.060185 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-171.ec2.internal" event="NodeHasSufficientPID" Nov 03 14:10:23.062350 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:23.062336 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-171.ec2.internal" Nov 03 14:10:23.062417 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:23.062359 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 14:10:23.063074 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:23.063046 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-171.ec2.internal" event="NodeHasSufficientMemory" Nov 03 14:10:23.063146 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:23.063089 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-171.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 14:10:23.063146 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:23.063100 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-171.ec2.internal" event="NodeHasSufficientPID" Nov 03 14:10:23.069404 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:23.069389 2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-171.ec2.internal" Nov 03 14:10:23.069473 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:23.069409 2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-171.ec2.internal\": node \"ip-10-0-143-171.ec2.internal\" not found" Nov 03 14:10:23.081578 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:23.081558 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/84a79a081fb234a3eba8ea0d2c88476d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal\" (UID: \"84a79a081fb234a3eba8ea0d2c88476d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal" Nov 03 14:10:23.081643 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:23.081583 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/84a79a081fb234a3eba8ea0d2c88476d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal\" (UID: \"84a79a081fb234a3eba8ea0d2c88476d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal" Nov 03 14:10:23.081643 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:23.081599 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3a1b8625ac3f3be5fcd1bbb72178f5c6-config\") pod \"kube-apiserver-proxy-ip-10-0-143-171.ec2.internal\" (UID: \"3a1b8625ac3f3be5fcd1bbb72178f5c6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-171.ec2.internal" Nov 03 14:10:23.093657 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:23.093641 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-171.ec2.internal\" not found" node="ip-10-0-143-171.ec2.internal" Nov 03 14:10:23.096600 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:23.096582 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-171.ec2.internal\" not found" node="ip-10-0-143-171.ec2.internal" Nov 03 14:10:23.110094 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:23.110078 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-171.ec2.internal\" not found" Nov 03 14:10:23.182539 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:23.182383 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3a1b8625ac3f3be5fcd1bbb72178f5c6-config\") pod \"kube-apiserver-proxy-ip-10-0-143-171.ec2.internal\" (UID: \"3a1b8625ac3f3be5fcd1bbb72178f5c6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-171.ec2.internal" Nov 03 14:10:23.182692 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:23.182474 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3a1b8625ac3f3be5fcd1bbb72178f5c6-config\") pod \"kube-apiserver-proxy-ip-10-0-143-171.ec2.internal\" (UID: \"3a1b8625ac3f3be5fcd1bbb72178f5c6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-171.ec2.internal" Nov 03 14:10:23.182692 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:23.182568 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/84a79a081fb234a3eba8ea0d2c88476d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal\" (UID: \"84a79a081fb234a3eba8ea0d2c88476d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal" Nov 03 14:10:23.182692 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:23.182546 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/84a79a081fb234a3eba8ea0d2c88476d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal\" (UID: \"84a79a081fb234a3eba8ea0d2c88476d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal" Nov 03 14:10:23.182692 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:23.182624 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/84a79a081fb234a3eba8ea0d2c88476d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal\" (UID: \"84a79a081fb234a3eba8ea0d2c88476d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal" Nov 03 14:10:23.182692 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:23.182655 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/84a79a081fb234a3eba8ea0d2c88476d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal\" (UID: \"84a79a081fb234a3eba8ea0d2c88476d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal" Nov 03 14:10:23.210434 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:23.210410 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-171.ec2.internal\" not found" Nov 03 14:10:23.311050 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:23.311030 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-171.ec2.internal\" not found" Nov 03 14:10:23.399145 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:23.399110 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal" Nov 03 14:10:23.401226 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:23.401211 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-171.ec2.internal" Nov 03 14:10:23.411119 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:23.411103 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-171.ec2.internal\" not found" Nov 03 14:10:23.512144 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:23.512092 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-171.ec2.internal\" not found" Nov 03 14:10:23.612615 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:23.612541 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-171.ec2.internal\" not found" Nov 03 14:10:23.713100 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:23.713047 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-171.ec2.internal\" not found" Nov 03 14:10:23.770216 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:23.770184 2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 14:10:25.731667 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:25.731645 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 14:10:25.732113 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:25.731780 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 14:10:25.732113 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:25.731780 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 14:10:25.744246 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:25.744218 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-171.ec2.internal\" not found" Nov 03 14:10:25.844800 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:25.844727 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-171.ec2.internal\" not found" Nov 03 14:10:25.859572 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:25.859532 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a1b8625ac3f3be5fcd1bbb72178f5c6.slice/crio-ddafe78a5f8133c3e0127472345a985b2211323b1eefc6f97145d0d119c24cd2 WatchSource:0}: Error finding container ddafe78a5f8133c3e0127472345a985b2211323b1eefc6f97145d0d119c24cd2: Status 404 returned error can't find the container with id ddafe78a5f8133c3e0127472345a985b2211323b1eefc6f97145d0d119c24cd2 Nov 03 14:10:25.860284 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:25.860257 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84a79a081fb234a3eba8ea0d2c88476d.slice/crio-f05d75fa7171d422ee7851daa8ca8d3543fdf8ecc1043d80c30e26d7ef0e46de WatchSource:0}: Error finding container f05d75fa7171d422ee7851daa8ca8d3543fdf8ecc1043d80c30e26d7ef0e46de: Status 404 returned error can't find the container with id f05d75fa7171d422ee7851daa8ca8d3543fdf8ecc1043d80c30e26d7ef0e46de Nov 03 14:10:25.866008 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:25.865983 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 14:10:25.889509 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:25.889464 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-171.ec2.internal" event={"ID":"3a1b8625ac3f3be5fcd1bbb72178f5c6","Type":"ContainerStarted","Data":"ddafe78a5f8133c3e0127472345a985b2211323b1eefc6f97145d0d119c24cd2"} Nov 03 14:10:25.890464 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:25.890445 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal" event={"ID":"84a79a081fb234a3eba8ea0d2c88476d","Type":"ContainerStarted","Data":"f05d75fa7171d422ee7851daa8ca8d3543fdf8ecc1043d80c30e26d7ef0e46de"} Nov 03 14:10:25.890876 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:25.890853 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 14:05:24 +0000 UTC" deadline="2028-04-01 12:42:51.910363544 +0000 UTC" Nov 03 14:10:25.890876 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:25.890875 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12358h32m26.019490996s" Nov 03 14:10:25.945538 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:25.945496 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-171.ec2.internal\" not found" Nov 03 14:10:25.963689 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:25.963653 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 14:10:25.964954 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:25.964938 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-171.ec2.internal" event="NodeHasSufficientMemory" Nov 03 14:10:25.965109 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:25.964965 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-171.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 14:10:25.965109 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:25.964975 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-171.ec2.internal" event="NodeHasSufficientPID" Nov 03 14:10:26.046467 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:26.046428 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-171.ec2.internal\" not found" Nov 03 14:10:26.048066 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.048054 2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 14:10:26.146729 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:26.146653 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-171.ec2.internal\" not found" Nov 03 14:10:26.247350 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:26.247309 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-171.ec2.internal\" not found" Nov 03 14:10:26.347939 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:26.347890 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-171.ec2.internal\" not found" Nov 03 14:10:26.440377 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.440287 2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 14:10:26.511126 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.511093 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal" Nov 03 14:10:26.540813 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.540780 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 14:10:26.542035 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.542012 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-171.ec2.internal" Nov 03 14:10:26.564387 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.564360 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 14:10:26.779611 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.779585 2577 apiserver.go:52] "Watching apiserver" Nov 03 14:10:26.810043 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.810018 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 14:10:26.824879 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.824850 2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 14:10:26.825279 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.825247 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-65nrx","openshift-ovn-kubernetes/ovnkube-node-w6d6z","kube-system/konnectivity-agent-dj8cg","kube-system/kube-apiserver-proxy-ip-10-0-143-171.ec2.internal","openshift-dns/node-resolver-q4gsb","openshift-multus/multus-additional-cni-plugins-kfkzp","openshift-network-diagnostics/network-check-target-xkhr9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz","openshift-image-registry/node-ca-t4b94","kube-system/global-pull-secret-syncer-2xrr4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal","openshift-multus/multus-q5cqm","openshift-cluster-node-tuning-operator/tuned-xcplz","openshift-multus/network-metrics-daemon-tc5nw"] Nov 03 14:10:26.827959 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.827939 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xrr4" Nov 03 14:10:26.828072 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:26.828018 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xrr4" podUID="ef43449c-b7dc-4e84-b785-63201f80d4da" Nov 03 14:10:26.830292 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.830277 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:26.832462 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.832439 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dj8cg" Nov 03 14:10:26.834603 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.834579 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc5nw" Nov 03 14:10:26.834715 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:26.834653 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc5nw" podUID="f52cae01-7951-4d9e-9a57-9ba5bcdf6857" Nov 03 14:10:26.836912 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.836893 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q4gsb" Nov 03 14:10:26.840163 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.839174 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kfkzp" Nov 03 14:10:26.841283 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.841263 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t4b94" Nov 03 14:10:26.844930 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.844910 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkhr9" Nov 03 14:10:26.845021 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:26.844972 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xkhr9" podUID="328dde79-0c85-47a3-b3f8-38d8cc8614c0" Nov 03 14:10:26.847392 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.847374 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:26.848231 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.848187 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 14:10:26.848231 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.848187 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 14:10:26.848347 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.848258 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 14:10:26.848347 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.848297 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2ft6v\"" Nov 03 14:10:26.848576 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.848545 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 14:10:26.849709 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.849686 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-65nrx" Nov 03 14:10:26.850069 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.850046 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-lqmd6\"" Nov 03 14:10:26.853043 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.853019 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 14:10:26.853043 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.853023 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-95wl7\"" Nov 03 14:10:26.853184 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.853047 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 14:10:26.855002 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.854985 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:10:26.855091 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.855058 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 14:10:26.855772 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.855736 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 14:10:26.855878 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.855778 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 14:10:26.856024 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.855916 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 14:10:26.856024 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.855984 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-q495v\"" Nov 03 14:10:26.857337 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.856151 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 14:10:26.857337 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.857039 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 14:10:26.857337 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.857140 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xm8pf\"" Nov 03 14:10:26.857337 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.857188 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 14:10:26.857337 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.857213 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 14:10:26.858544 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.858525 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-q5cqm" Nov 03 14:10:26.867785 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.867765 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 14:10:26.867896 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.867828 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-89m6k" Nov 03 14:10:26.877725 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.877706 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-89m6k" Nov 03 14:10:26.911981 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.911960 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 14:10:26.925290 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.925255 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-etc-modprobe-d\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:26.925391 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.925290 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-run\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:26.925391 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.925317 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kfkzp\" (UID: \"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd\") " pod="openshift-multus/multus-additional-cni-plugins-kfkzp" Nov 03 14:10:26.925391 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.925344 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d533df4c-38b2-42b2-9cb2-b82afac33795-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zkvrz\" (UID: \"d533df4c-38b2-42b2-9cb2-b82afac33795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:10:26.925391 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.925369 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d533df4c-38b2-42b2-9cb2-b82afac33795-registration-dir\") pod \"aws-ebs-csi-driver-node-zkvrz\" (UID: \"d533df4c-38b2-42b2-9cb2-b82afac33795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:10:26.925553 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.925394 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-host-var-lib-kubelet\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:26.925553 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.925417 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d533df4c-38b2-42b2-9cb2-b82afac33795-etc-selinux\") pod \"aws-ebs-csi-driver-node-zkvrz\" (UID: \"d533df4c-38b2-42b2-9cb2-b82afac33795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:10:26.925553 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.925439 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-host-run-netns\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:26.925553 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.925463 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-host-cni-netd\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:26.925553 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.925485 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-host-var-lib-cni-bin\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:26.925553 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.925507 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-systemd-units\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:26.925553 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.925529 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-host-run-netns\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:26.925553 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.925550 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-run-ovn\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:26.925873 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.925576 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-ovnkube-config\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:26.925873 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.925606 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-lib-modules\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:26.925873 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.925631 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6kdm\" (UniqueName: \"kubernetes.io/projected/a6466fc4-ed3d-4031-ba11-5606bd1af74b-kube-api-access-s6kdm\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:26.925873 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.925655 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd-cni-binary-copy\") pod \"multus-additional-cni-plugins-kfkzp\" (UID: \"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd\") " pod="openshift-multus/multus-additional-cni-plugins-kfkzp" Nov 03 14:10:26.925873 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.925680 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-multus-conf-dir\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:26.925873 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.925703 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e9dee2ef-6437-4963-8e27-da5fb9c75928-agent-certs\") pod \"konnectivity-agent-dj8cg\" (UID: \"e9dee2ef-6437-4963-8e27-da5fb9c75928\") " pod="kube-system/konnectivity-agent-dj8cg" Nov 03 14:10:26.925873 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.925729 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-host-slash\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:26.925873 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.925751 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-env-overrides\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:26.925873 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.925775 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-ovn-node-metrics-cert\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:26.925873 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.925798 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ef43449c-b7dc-4e84-b785-63201f80d4da-kubelet-config\") pod \"global-pull-secret-syncer-2xrr4\" (UID: \"ef43449c-b7dc-4e84-b785-63201f80d4da\") " pod="kube-system/global-pull-secret-syncer-2xrr4" Nov 03 14:10:26.925873 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.925821 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-hostroot\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:26.925873 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.925860 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3fe332c5-8991-4d94-9b67-c9017f4945b8-multus-daemon-config\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:26.926326 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.925892 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkrl8\" (UniqueName: \"kubernetes.io/projected/f52cae01-7951-4d9e-9a57-9ba5bcdf6857-kube-api-access-fkrl8\") pod \"network-metrics-daemon-tc5nw\" (UID: \"f52cae01-7951-4d9e-9a57-9ba5bcdf6857\") " pod="openshift-multus/network-metrics-daemon-tc5nw" Nov 03 14:10:26.926326 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.925916 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-etc-openvswitch\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:26.926326 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.925938 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-host-cni-bin\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:26.926326 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.925962 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7782\" (UniqueName: \"kubernetes.io/projected/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-kube-api-access-g7782\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:26.926326 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.925997 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/396e7be7-0e72-4c32-a04a-22bd9bd708e1-host-slash\") pod \"iptables-alerter-65nrx\" (UID: \"396e7be7-0e72-4c32-a04a-22bd9bd708e1\") " pod="openshift-network-operator/iptables-alerter-65nrx" Nov 03 14:10:26.926326 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.926026 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef43449c-b7dc-4e84-b785-63201f80d4da-original-pull-secret\") pod \"global-pull-secret-syncer-2xrr4\" (UID: \"ef43449c-b7dc-4e84-b785-63201f80d4da\") " pod="kube-system/global-pull-secret-syncer-2xrr4" Nov 03 14:10:26.926326 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.926051 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkjrd\" (UniqueName: \"kubernetes.io/projected/328dde79-0c85-47a3-b3f8-38d8cc8614c0-kube-api-access-dkjrd\") pod \"network-check-target-xkhr9\" (UID: \"328dde79-0c85-47a3-b3f8-38d8cc8614c0\") " pod="openshift-network-diagnostics/network-check-target-xkhr9" Nov 03 14:10:26.926326 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.926075 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d533df4c-38b2-42b2-9cb2-b82afac33795-socket-dir\") pod \"aws-ebs-csi-driver-node-zkvrz\" (UID: \"d533df4c-38b2-42b2-9cb2-b82afac33795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:10:26.926326 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.926098 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-etc-sysctl-d\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:26.926326 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.926125 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-sys\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:26.926326 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.926149 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-var-lib-kubelet\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:26.926326 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.926173 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kfkzp\" (UID: \"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd\") " pod="openshift-multus/multus-additional-cni-plugins-kfkzp" Nov 03 14:10:26.926326 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.926201 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e637e650-5fc6-4adb-a97a-47d924c30f97-tmp-dir\") pod \"node-resolver-q4gsb\" (UID: \"e637e650-5fc6-4adb-a97a-47d924c30f97\") " pod="openshift-dns/node-resolver-q4gsb" Nov 03 14:10:26.926326 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.926226 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-ovnkube-script-lib\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:26.926326 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.926256 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e9dee2ef-6437-4963-8e27-da5fb9c75928-konnectivity-ca\") pod \"konnectivity-agent-dj8cg\" (UID: \"e9dee2ef-6437-4963-8e27-da5fb9c75928\") " pod="kube-system/konnectivity-agent-dj8cg" Nov 03 14:10:26.926326 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.926279 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f52cae01-7951-4d9e-9a57-9ba5bcdf6857-metrics-certs\") pod \"network-metrics-daemon-tc5nw\" (UID: \"f52cae01-7951-4d9e-9a57-9ba5bcdf6857\") " pod="openshift-multus/network-metrics-daemon-tc5nw" Nov 03 14:10:26.926966 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.926302 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e637e650-5fc6-4adb-a97a-47d924c30f97-hosts-file\") pod \"node-resolver-q4gsb\" (UID: \"e637e650-5fc6-4adb-a97a-47d924c30f97\") " pod="openshift-dns/node-resolver-q4gsb" Nov 03 14:10:26.926966 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.926331 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-etc-systemd\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:26.926966 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.926363 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d533df4c-38b2-42b2-9cb2-b82afac33795-device-dir\") pod \"aws-ebs-csi-driver-node-zkvrz\" (UID: \"d533df4c-38b2-42b2-9cb2-b82afac33795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:10:26.926966 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.926387 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl2f4\" (UniqueName: \"kubernetes.io/projected/3fe332c5-8991-4d94-9b67-c9017f4945b8-kube-api-access-xl2f4\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:26.926966 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.926411 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-log-socket\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:26.926966 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.926434 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a6466fc4-ed3d-4031-ba11-5606bd1af74b-etc-tuned\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:26.926966 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.926457 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6466fc4-ed3d-4031-ba11-5606bd1af74b-tmp\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:26.926966 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.926480 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3fe332c5-8991-4d94-9b67-c9017f4945b8-cni-binary-copy\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:26.926966 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.926503 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-host-run-k8s-cni-cncf-io\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:26.926966 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.926529 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/396e7be7-0e72-4c32-a04a-22bd9bd708e1-iptables-alerter-script\") pod \"iptables-alerter-65nrx\" (UID: \"396e7be7-0e72-4c32-a04a-22bd9bd708e1\") " pod="openshift-network-operator/iptables-alerter-65nrx" Nov 03 14:10:26.926966 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.926554 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmwwz\" (UniqueName: \"kubernetes.io/projected/396e7be7-0e72-4c32-a04a-22bd9bd708e1-kube-api-access-fmwwz\") pod \"iptables-alerter-65nrx\" (UID: \"396e7be7-0e72-4c32-a04a-22bd9bd708e1\") " pod="openshift-network-operator/iptables-alerter-65nrx" Nov 03 14:10:26.926966 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.926581 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-system-cni-dir\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:26.926966 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.926606 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-host-kubelet\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:26.926966 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.926631 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-run-openvswitch\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:26.926966 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.926655 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-run-systemd\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:26.926966 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.926681 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp5h5\" (UniqueName: \"kubernetes.io/projected/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd-kube-api-access-vp5h5\") pod \"multus-additional-cni-plugins-kfkzp\" (UID: \"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd\") " pod="openshift-multus/multus-additional-cni-plugins-kfkzp" Nov 03 14:10:26.927597 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.926705 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-etc-kubernetes\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:26.927597 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.926728 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-etc-sysctl-conf\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:26.927597 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.926752 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-multus-cni-dir\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:26.927597 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.926798 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-multus-socket-dir-parent\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:26.927597 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.926828 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-host-var-lib-cni-multus\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:26.927597 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.926867 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ef43449c-b7dc-4e84-b785-63201f80d4da-dbus\") pod \"global-pull-secret-syncer-2xrr4\" (UID: \"ef43449c-b7dc-4e84-b785-63201f80d4da\") " pod="kube-system/global-pull-secret-syncer-2xrr4" Nov 03 14:10:26.927597 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.926899 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-etc-sysconfig\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:26.927597 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.926923 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-host\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:26.927597 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.926947 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd-cnibin\") pod \"multus-additional-cni-plugins-kfkzp\" (UID: \"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd\") " pod="openshift-multus/multus-additional-cni-plugins-kfkzp" Nov 03 14:10:26.927597 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.926970 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d533df4c-38b2-42b2-9cb2-b82afac33795-sys-fs\") pod \"aws-ebs-csi-driver-node-zkvrz\" (UID: \"d533df4c-38b2-42b2-9cb2-b82afac33795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:10:26.927597 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.926996 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr7zc\" (UniqueName: \"kubernetes.io/projected/e637e650-5fc6-4adb-a97a-47d924c30f97-kube-api-access-mr7zc\") pod \"node-resolver-q4gsb\" (UID: \"e637e650-5fc6-4adb-a97a-47d924c30f97\") " pod="openshift-dns/node-resolver-q4gsb" Nov 03 14:10:26.927597 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.927027 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e68be0c4-d4ee-4da5-82e3-82372f811841-serviceca\") pod \"node-ca-t4b94\" (UID: \"e68be0c4-d4ee-4da5-82e3-82372f811841\") " pod="openshift-image-registry/node-ca-t4b94" Nov 03 14:10:26.927597 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.927054 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-var-lib-openvswitch\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:26.927597 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.927083 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e68be0c4-d4ee-4da5-82e3-82372f811841-host\") pod \"node-ca-t4b94\" (UID: \"e68be0c4-d4ee-4da5-82e3-82372f811841\") " pod="openshift-image-registry/node-ca-t4b94" Nov 03 14:10:26.927597 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.927111 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-node-log\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:26.927597 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.927138 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:26.928181 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.927167 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfv25\" (UniqueName: \"kubernetes.io/projected/d533df4c-38b2-42b2-9cb2-b82afac33795-kube-api-access-sfv25\") pod \"aws-ebs-csi-driver-node-zkvrz\" (UID: \"d533df4c-38b2-42b2-9cb2-b82afac33795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:10:26.928181 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.927193 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-etc-kubernetes\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:26.928181 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.927220 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgm54\" (UniqueName: \"kubernetes.io/projected/e68be0c4-d4ee-4da5-82e3-82372f811841-kube-api-access-wgm54\") pod \"node-ca-t4b94\" (UID: \"e68be0c4-d4ee-4da5-82e3-82372f811841\") " pod="openshift-image-registry/node-ca-t4b94" Nov 03 14:10:26.928181 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.927245 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-host-run-ovn-kubernetes\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:26.928181 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.927285 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kfkzp\" (UID: \"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd\") " pod="openshift-multus/multus-additional-cni-plugins-kfkzp" Nov 03 14:10:26.928181 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.927311 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd-system-cni-dir\") pod \"multus-additional-cni-plugins-kfkzp\" (UID: \"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd\") " pod="openshift-multus/multus-additional-cni-plugins-kfkzp" Nov 03 14:10:26.928181 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.927337 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-cnibin\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:26.928181 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.927363 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-host-run-multus-certs\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:26.948051 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.948028 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 14:10:26.948151 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.948036 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 14:10:26.948151 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.948072 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 14:10:26.948151 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.948041 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 14:10:26.948151 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.948134 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 14:10:26.948324 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.948164 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 14:10:26.948324 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.948215 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 14:10:26.948324 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.948229 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 14:10:26.948324 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.948243 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qhrll\"" Nov 03 14:10:26.948511 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.948443 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-tb44c\"" Nov 03 14:10:26.948511 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.948497 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-trbj6\"" Nov 03 14:10:26.948602 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.948499 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 14:10:26.948702 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.948688 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 14:10:26.948787 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.948762 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 14:10:26.948787 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.948763 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 14:10:26.948909 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.948798 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qgxkz\"" Nov 03 14:10:26.948962 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:26.948953 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 14:10:27.027695 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.027662 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d533df4c-38b2-42b2-9cb2-b82afac33795-registration-dir\") pod \"aws-ebs-csi-driver-node-zkvrz\" (UID: \"d533df4c-38b2-42b2-9cb2-b82afac33795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:10:27.027695 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.027699 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-host-var-lib-kubelet\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:27.027994 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.027723 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d533df4c-38b2-42b2-9cb2-b82afac33795-etc-selinux\") pod \"aws-ebs-csi-driver-node-zkvrz\" (UID: \"d533df4c-38b2-42b2-9cb2-b82afac33795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:10:27.027994 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.027745 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-host-run-netns\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:27.027994 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.027765 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-host-cni-netd\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:27.027994 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.027787 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-host-var-lib-cni-bin\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:27.027994 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.027810 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-systemd-units\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:27.027994 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.027845 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-host-run-netns\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:27.027994 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.027867 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-host-run-netns\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:27.027994 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.027870 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-run-ovn\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:27.027994 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.027913 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-run-ovn\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:27.027994 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.027917 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-host-var-lib-cni-bin\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:27.027994 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.027805 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d533df4c-38b2-42b2-9cb2-b82afac33795-registration-dir\") pod \"aws-ebs-csi-driver-node-zkvrz\" (UID: \"d533df4c-38b2-42b2-9cb2-b82afac33795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:10:27.027994 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.027926 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-ovnkube-config\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:27.027994 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.027979 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-host-var-lib-kubelet\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:27.028604 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.028024 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-host-run-netns\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:27.028604 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.028053 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-lib-modules\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:27.028604 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.028056 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-systemd-units\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:27.028604 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.028077 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6kdm\" (UniqueName: \"kubernetes.io/projected/a6466fc4-ed3d-4031-ba11-5606bd1af74b-kube-api-access-s6kdm\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:27.028604 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.028096 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-host-cni-netd\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:27.028604 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.028137 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd-cni-binary-copy\") pod \"multus-additional-cni-plugins-kfkzp\" (UID: \"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd\") " pod="openshift-multus/multus-additional-cni-plugins-kfkzp" Nov 03 14:10:27.028604 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.028167 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-multus-conf-dir\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:27.028604 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.028194 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e9dee2ef-6437-4963-8e27-da5fb9c75928-agent-certs\") pod \"konnectivity-agent-dj8cg\" (UID: \"e9dee2ef-6437-4963-8e27-da5fb9c75928\") " pod="kube-system/konnectivity-agent-dj8cg" Nov 03 14:10:27.028604 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.028320 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-host-slash\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:27.028604 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.028343 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-env-overrides\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:27.028604 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.028375 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-ovn-node-metrics-cert\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:27.028604 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.028394 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ef43449c-b7dc-4e84-b785-63201f80d4da-kubelet-config\") pod \"global-pull-secret-syncer-2xrr4\" (UID: \"ef43449c-b7dc-4e84-b785-63201f80d4da\") " pod="kube-system/global-pull-secret-syncer-2xrr4" Nov 03 14:10:27.028604 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.028410 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-hostroot\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:27.028604 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.028425 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3fe332c5-8991-4d94-9b67-c9017f4945b8-multus-daemon-config\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:27.028604 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.028500 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkrl8\" (UniqueName: \"kubernetes.io/projected/f52cae01-7951-4d9e-9a57-9ba5bcdf6857-kube-api-access-fkrl8\") pod \"network-metrics-daemon-tc5nw\" (UID: \"f52cae01-7951-4d9e-9a57-9ba5bcdf6857\") " pod="openshift-multus/network-metrics-daemon-tc5nw" Nov 03 14:10:27.028604 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.028515 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-etc-openvswitch\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:27.028604 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.028536 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-host-cni-bin\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:27.029364 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.028557 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7782\" (UniqueName: \"kubernetes.io/projected/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-kube-api-access-g7782\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:27.029364 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.028573 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/396e7be7-0e72-4c32-a04a-22bd9bd708e1-host-slash\") pod \"iptables-alerter-65nrx\" (UID: \"396e7be7-0e72-4c32-a04a-22bd9bd708e1\") " pod="openshift-network-operator/iptables-alerter-65nrx" Nov 03 14:10:27.029364 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.028577 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-ovnkube-config\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:27.029364 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.028594 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef43449c-b7dc-4e84-b785-63201f80d4da-original-pull-secret\") pod \"global-pull-secret-syncer-2xrr4\" (UID: \"ef43449c-b7dc-4e84-b785-63201f80d4da\") " pod="kube-system/global-pull-secret-syncer-2xrr4" Nov 03 14:10:27.029364 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.028622 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkjrd\" (UniqueName: \"kubernetes.io/projected/328dde79-0c85-47a3-b3f8-38d8cc8614c0-kube-api-access-dkjrd\") pod \"network-check-target-xkhr9\" (UID: \"328dde79-0c85-47a3-b3f8-38d8cc8614c0\") " pod="openshift-network-diagnostics/network-check-target-xkhr9" Nov 03 14:10:27.029364 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.028637 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d533df4c-38b2-42b2-9cb2-b82afac33795-socket-dir\") pod \"aws-ebs-csi-driver-node-zkvrz\" (UID: \"d533df4c-38b2-42b2-9cb2-b82afac33795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:10:27.029364 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.028654 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-etc-sysctl-d\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:27.029364 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.028669 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-sys\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:27.029364 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.028692 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-var-lib-kubelet\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:27.029364 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.028736 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd-cni-binary-copy\") pod \"multus-additional-cni-plugins-kfkzp\" (UID: \"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd\") " pod="openshift-multus/multus-additional-cni-plugins-kfkzp" Nov 03 14:10:27.029364 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.028789 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kfkzp\" (UID: \"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd\") " pod="openshift-multus/multus-additional-cni-plugins-kfkzp" Nov 03 14:10:27.029364 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.028815 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e637e650-5fc6-4adb-a97a-47d924c30f97-tmp-dir\") pod \"node-resolver-q4gsb\" (UID: \"e637e650-5fc6-4adb-a97a-47d924c30f97\") " pod="openshift-dns/node-resolver-q4gsb" Nov 03 14:10:27.029364 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.028848 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-ovnkube-script-lib\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:27.029364 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.028868 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e9dee2ef-6437-4963-8e27-da5fb9c75928-konnectivity-ca\") pod \"konnectivity-agent-dj8cg\" (UID: \"e9dee2ef-6437-4963-8e27-da5fb9c75928\") " pod="kube-system/konnectivity-agent-dj8cg" Nov 03 14:10:27.029364 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.028894 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f52cae01-7951-4d9e-9a57-9ba5bcdf6857-metrics-certs\") pod \"network-metrics-daemon-tc5nw\" (UID: \"f52cae01-7951-4d9e-9a57-9ba5bcdf6857\") " pod="openshift-multus/network-metrics-daemon-tc5nw" Nov 03 14:10:27.029364 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.028903 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-etc-sysctl-d\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:27.029364 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.028918 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e637e650-5fc6-4adb-a97a-47d924c30f97-hosts-file\") pod \"node-resolver-q4gsb\" (UID: \"e637e650-5fc6-4adb-a97a-47d924c30f97\") " pod="openshift-dns/node-resolver-q4gsb" Nov 03 14:10:27.030226 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.028929 2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 14:10:27.030226 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.028946 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-etc-systemd\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:27.030226 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.028969 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d533df4c-38b2-42b2-9cb2-b82afac33795-device-dir\") pod \"aws-ebs-csi-driver-node-zkvrz\" (UID: \"d533df4c-38b2-42b2-9cb2-b82afac33795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:10:27.030226 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:27.028989 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 14:10:27.030226 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.028993 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl2f4\" (UniqueName: \"kubernetes.io/projected/3fe332c5-8991-4d94-9b67-c9017f4945b8-kube-api-access-xl2f4\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:27.030226 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.029023 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-log-socket\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:27.030226 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:27.029052 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef43449c-b7dc-4e84-b785-63201f80d4da-original-pull-secret podName:ef43449c-b7dc-4e84-b785-63201f80d4da nodeName:}" failed. No retries permitted until 2026-11-03 14:10:27.529032288 +0000 UTC m=+3.145158897 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef43449c-b7dc-4e84-b785-63201f80d4da-original-pull-secret") pod "global-pull-secret-syncer-2xrr4" (UID: "ef43449c-b7dc-4e84-b785-63201f80d4da") : object "kube-system"/"original-pull-secret" not registered Nov 03 14:10:27.030226 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.029068 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-log-socket\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:27.030226 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.029083 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a6466fc4-ed3d-4031-ba11-5606bd1af74b-etc-tuned\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:27.030226 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.029111 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6466fc4-ed3d-4031-ba11-5606bd1af74b-tmp\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:27.030226 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.029136 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3fe332c5-8991-4d94-9b67-c9017f4945b8-cni-binary-copy\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:27.030226 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.029164 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-host-run-k8s-cni-cncf-io\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:27.030226 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.029190 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/396e7be7-0e72-4c32-a04a-22bd9bd708e1-iptables-alerter-script\") pod \"iptables-alerter-65nrx\" (UID: \"396e7be7-0e72-4c32-a04a-22bd9bd708e1\") " pod="openshift-network-operator/iptables-alerter-65nrx" Nov 03 14:10:27.030226 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.029215 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmwwz\" (UniqueName: \"kubernetes.io/projected/396e7be7-0e72-4c32-a04a-22bd9bd708e1-kube-api-access-fmwwz\") pod \"iptables-alerter-65nrx\" (UID: \"396e7be7-0e72-4c32-a04a-22bd9bd708e1\") " pod="openshift-network-operator/iptables-alerter-65nrx" Nov 03 14:10:27.030226 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.029391 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ef43449c-b7dc-4e84-b785-63201f80d4da-kubelet-config\") pod \"global-pull-secret-syncer-2xrr4\" (UID: \"ef43449c-b7dc-4e84-b785-63201f80d4da\") " pod="kube-system/global-pull-secret-syncer-2xrr4" Nov 03 14:10:27.030226 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.029431 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-hostroot\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:27.030226 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.029481 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d533df4c-38b2-42b2-9cb2-b82afac33795-device-dir\") pod \"aws-ebs-csi-driver-node-zkvrz\" (UID: \"d533df4c-38b2-42b2-9cb2-b82afac33795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:10:27.030889 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.029546 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d533df4c-38b2-42b2-9cb2-b82afac33795-socket-dir\") pod \"aws-ebs-csi-driver-node-zkvrz\" (UID: \"d533df4c-38b2-42b2-9cb2-b82afac33795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:10:27.030889 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.029697 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-etc-openvswitch\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:27.030889 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.029745 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-sys\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:27.030889 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.029786 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-etc-systemd\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:27.030889 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.029825 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e9dee2ef-6437-4963-8e27-da5fb9c75928-konnectivity-ca\") pod \"konnectivity-agent-dj8cg\" (UID: \"e9dee2ef-6437-4963-8e27-da5fb9c75928\") " pod="kube-system/konnectivity-agent-dj8cg" Nov 03 14:10:27.030889 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:27.029869 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 14:10:27.030889 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:27.029922 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f52cae01-7951-4d9e-9a57-9ba5bcdf6857-metrics-certs podName:f52cae01-7951-4d9e-9a57-9ba5bcdf6857 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:27.529903636 +0000 UTC m=+3.146030239 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f52cae01-7951-4d9e-9a57-9ba5bcdf6857-metrics-certs") pod "network-metrics-daemon-tc5nw" (UID: "f52cae01-7951-4d9e-9a57-9ba5bcdf6857") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 14:10:27.030889 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.029969 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e637e650-5fc6-4adb-a97a-47d924c30f97-hosts-file\") pod \"node-resolver-q4gsb\" (UID: \"e637e650-5fc6-4adb-a97a-47d924c30f97\") " pod="openshift-dns/node-resolver-q4gsb" Nov 03 14:10:27.030889 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.030014 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d533df4c-38b2-42b2-9cb2-b82afac33795-etc-selinux\") pod \"aws-ebs-csi-driver-node-zkvrz\" (UID: \"d533df4c-38b2-42b2-9cb2-b82afac33795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:10:27.031246 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.030918 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3fe332c5-8991-4d94-9b67-c9017f4945b8-cni-binary-copy\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:27.031246 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.030970 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-host-cni-bin\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:27.031246 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.031175 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/396e7be7-0e72-4c32-a04a-22bd9bd708e1-host-slash\") pod \"iptables-alerter-65nrx\" (UID: \"396e7be7-0e72-4c32-a04a-22bd9bd708e1\") " pod="openshift-network-operator/iptables-alerter-65nrx" Nov 03 14:10:27.031572 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.031547 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kfkzp\" (UID: \"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd\") " pod="openshift-multus/multus-additional-cni-plugins-kfkzp" Nov 03 14:10:27.031850 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.031799 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e637e650-5fc6-4adb-a97a-47d924c30f97-tmp-dir\") pod \"node-resolver-q4gsb\" (UID: \"e637e650-5fc6-4adb-a97a-47d924c30f97\") " pod="openshift-dns/node-resolver-q4gsb" Nov 03 14:10:27.032751 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032107 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-host-slash\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:27.032751 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032117 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-multus-conf-dir\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:27.032751 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032132 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-var-lib-kubelet\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:27.032751 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032142 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-lib-modules\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:27.032751 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032170 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-host-run-k8s-cni-cncf-io\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:27.032751 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032176 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-system-cni-dir\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:27.032751 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032198 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-host-kubelet\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:27.032751 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032219 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-run-openvswitch\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:27.032751 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032245 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-run-systemd\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:27.032751 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032270 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp5h5\" (UniqueName: \"kubernetes.io/projected/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd-kube-api-access-vp5h5\") pod \"multus-additional-cni-plugins-kfkzp\" (UID: \"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd\") " pod="openshift-multus/multus-additional-cni-plugins-kfkzp" Nov 03 14:10:27.032751 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032293 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-etc-kubernetes\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:27.032751 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032299 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-ovnkube-script-lib\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:27.032751 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032309 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-etc-sysctl-conf\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:27.032751 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032336 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-multus-cni-dir\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:27.032751 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032351 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-multus-socket-dir-parent\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:27.032751 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032373 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-host-var-lib-cni-multus\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:27.032751 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032371 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-run-systemd\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:27.032751 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032388 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ef43449c-b7dc-4e84-b785-63201f80d4da-dbus\") pod \"global-pull-secret-syncer-2xrr4\" (UID: \"ef43449c-b7dc-4e84-b785-63201f80d4da\") " pod="kube-system/global-pull-secret-syncer-2xrr4" Nov 03 14:10:27.033672 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032425 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-etc-sysconfig\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:27.033672 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032444 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-host\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:27.033672 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032474 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-etc-kubernetes\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:27.033672 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032482 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd-cnibin\") pod \"multus-additional-cni-plugins-kfkzp\" (UID: \"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd\") " pod="openshift-multus/multus-additional-cni-plugins-kfkzp" Nov 03 14:10:27.033672 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032505 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-env-overrides\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:27.033672 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032531 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd-cnibin\") pod \"multus-additional-cni-plugins-kfkzp\" (UID: \"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd\") " pod="openshift-multus/multus-additional-cni-plugins-kfkzp" Nov 03 14:10:27.033672 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032555 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-run-openvswitch\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:27.033672 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032581 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d533df4c-38b2-42b2-9cb2-b82afac33795-sys-fs\") pod \"aws-ebs-csi-driver-node-zkvrz\" (UID: \"d533df4c-38b2-42b2-9cb2-b82afac33795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:10:27.033672 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032609 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr7zc\" (UniqueName: \"kubernetes.io/projected/e637e650-5fc6-4adb-a97a-47d924c30f97-kube-api-access-mr7zc\") pod \"node-resolver-q4gsb\" (UID: \"e637e650-5fc6-4adb-a97a-47d924c30f97\") " pod="openshift-dns/node-resolver-q4gsb" Nov 03 14:10:27.033672 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032624 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ef43449c-b7dc-4e84-b785-63201f80d4da-dbus\") pod \"global-pull-secret-syncer-2xrr4\" (UID: \"ef43449c-b7dc-4e84-b785-63201f80d4da\") " pod="kube-system/global-pull-secret-syncer-2xrr4" Nov 03 14:10:27.033672 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032636 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e68be0c4-d4ee-4da5-82e3-82372f811841-serviceca\") pod \"node-ca-t4b94\" (UID: \"e68be0c4-d4ee-4da5-82e3-82372f811841\") " pod="openshift-image-registry/node-ca-t4b94" Nov 03 14:10:27.033672 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032643 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-multus-cni-dir\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:27.033672 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032655 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-etc-sysconfig\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:27.033672 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032663 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-var-lib-openvswitch\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:27.033672 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032664 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d533df4c-38b2-42b2-9cb2-b82afac33795-sys-fs\") pod \"aws-ebs-csi-driver-node-zkvrz\" (UID: \"d533df4c-38b2-42b2-9cb2-b82afac33795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:10:27.033672 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032684 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-etc-sysctl-conf\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:27.033672 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032693 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-host-kubelet\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:27.033672 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032692 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e68be0c4-d4ee-4da5-82e3-82372f811841-host\") pod \"node-ca-t4b94\" (UID: \"e68be0c4-d4ee-4da5-82e3-82372f811841\") " pod="openshift-image-registry/node-ca-t4b94" Nov 03 14:10:27.034296 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032677 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-host-var-lib-cni-multus\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:27.034296 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032708 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-multus-socket-dir-parent\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:27.034296 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032737 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-node-log\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:27.034296 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032750 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-system-cni-dir\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:27.034296 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032763 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:27.034296 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032787 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-var-lib-openvswitch\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:27.034296 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032791 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfv25\" (UniqueName: \"kubernetes.io/projected/d533df4c-38b2-42b2-9cb2-b82afac33795-kube-api-access-sfv25\") pod \"aws-ebs-csi-driver-node-zkvrz\" (UID: \"d533df4c-38b2-42b2-9cb2-b82afac33795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:10:27.034296 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032848 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-etc-kubernetes\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:27.034296 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032855 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:27.034296 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032737 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-host\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:27.034296 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032875 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgm54\" (UniqueName: \"kubernetes.io/projected/e68be0c4-d4ee-4da5-82e3-82372f811841-kube-api-access-wgm54\") pod \"node-ca-t4b94\" (UID: \"e68be0c4-d4ee-4da5-82e3-82372f811841\") " pod="openshift-image-registry/node-ca-t4b94" Nov 03 14:10:27.034296 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032901 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-host-run-ovn-kubernetes\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:27.034296 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.032903 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-node-log\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:27.034296 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.033145 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e68be0c4-d4ee-4da5-82e3-82372f811841-host\") pod \"node-ca-t4b94\" (UID: \"e68be0c4-d4ee-4da5-82e3-82372f811841\") " pod="openshift-image-registry/node-ca-t4b94" Nov 03 14:10:27.034296 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.033186 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kfkzp\" (UID: \"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd\") " pod="openshift-multus/multus-additional-cni-plugins-kfkzp" Nov 03 14:10:27.034296 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.033214 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd-system-cni-dir\") pod \"multus-additional-cni-plugins-kfkzp\" (UID: \"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd\") " pod="openshift-multus/multus-additional-cni-plugins-kfkzp" Nov 03 14:10:27.034296 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.033253 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-cnibin\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:27.034738 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.033281 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-host-run-multus-certs\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:27.034738 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.033329 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-etc-modprobe-d\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:27.034738 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.033348 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-run\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:27.034738 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.033364 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kfkzp\" (UID: \"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd\") " pod="openshift-multus/multus-additional-cni-plugins-kfkzp" Nov 03 14:10:27.034738 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.033384 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d533df4c-38b2-42b2-9cb2-b82afac33795-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zkvrz\" (UID: \"d533df4c-38b2-42b2-9cb2-b82afac33795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:10:27.034738 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.033482 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d533df4c-38b2-42b2-9cb2-b82afac33795-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zkvrz\" (UID: \"d533df4c-38b2-42b2-9cb2-b82afac33795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:10:27.034738 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.033519 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-host-run-multus-certs\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:27.034738 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.033550 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-etc-kubernetes\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:27.034738 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.033733 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-host-run-ovn-kubernetes\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:27.034738 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.033947 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e68be0c4-d4ee-4da5-82e3-82372f811841-serviceca\") pod \"node-ca-t4b94\" (UID: \"e68be0c4-d4ee-4da5-82e3-82372f811841\") " pod="openshift-image-registry/node-ca-t4b94" Nov 03 14:10:27.034738 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.034034 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-etc-modprobe-d\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:27.034738 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.034061 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd-system-cni-dir\") pod \"multus-additional-cni-plugins-kfkzp\" (UID: \"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd\") " pod="openshift-multus/multus-additional-cni-plugins-kfkzp" Nov 03 14:10:27.034738 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.034094 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-cnibin\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:27.034738 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.034247 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/396e7be7-0e72-4c32-a04a-22bd9bd708e1-iptables-alerter-script\") pod \"iptables-alerter-65nrx\" (UID: \"396e7be7-0e72-4c32-a04a-22bd9bd708e1\") " pod="openshift-network-operator/iptables-alerter-65nrx" Nov 03 14:10:27.034738 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.034302 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-run\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:27.034738 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.034247 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3fe332c5-8991-4d94-9b67-c9017f4945b8-multus-daemon-config\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:27.034738 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.034396 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-ovn-node-metrics-cert\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:27.035481 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.034662 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kfkzp\" (UID: \"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd\") " pod="openshift-multus/multus-additional-cni-plugins-kfkzp" Nov 03 14:10:27.036297 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.036242 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e9dee2ef-6437-4963-8e27-da5fb9c75928-agent-certs\") pod \"konnectivity-agent-dj8cg\" (UID: \"e9dee2ef-6437-4963-8e27-da5fb9c75928\") " pod="kube-system/konnectivity-agent-dj8cg" Nov 03 14:10:27.036788 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.036764 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a6466fc4-ed3d-4031-ba11-5606bd1af74b-etc-tuned\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:27.038263 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.038237 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6466fc4-ed3d-4031-ba11-5606bd1af74b-tmp\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:27.041008 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.040987 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kfkzp\" (UID: \"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd\") " pod="openshift-multus/multus-additional-cni-plugins-kfkzp" Nov 03 14:10:27.065908 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:27.065878 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 14:10:27.065908 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:27.065908 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 14:10:27.066104 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:27.065924 2577 projected.go:196] Error preparing data for projected volume kube-api-access-dkjrd for pod openshift-network-diagnostics/network-check-target-xkhr9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 14:10:27.066104 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:27.066029 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/328dde79-0c85-47a3-b3f8-38d8cc8614c0-kube-api-access-dkjrd podName:328dde79-0c85-47a3-b3f8-38d8cc8614c0 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:27.566007634 +0000 UTC m=+3.182134216 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dkjrd" (UniqueName: "kubernetes.io/projected/328dde79-0c85-47a3-b3f8-38d8cc8614c0-kube-api-access-dkjrd") pod "network-check-target-xkhr9" (UID: "328dde79-0c85-47a3-b3f8-38d8cc8614c0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 14:10:27.068760 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.068676 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl2f4\" (UniqueName: \"kubernetes.io/projected/3fe332c5-8991-4d94-9b67-c9017f4945b8-kube-api-access-xl2f4\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:10:27.068944 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.068911 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmwwz\" (UniqueName: \"kubernetes.io/projected/396e7be7-0e72-4c32-a04a-22bd9bd708e1-kube-api-access-fmwwz\") pod \"iptables-alerter-65nrx\" (UID: \"396e7be7-0e72-4c32-a04a-22bd9bd708e1\") " pod="openshift-network-operator/iptables-alerter-65nrx" Nov 03 14:10:27.070169 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.070137 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr7zc\" (UniqueName: \"kubernetes.io/projected/e637e650-5fc6-4adb-a97a-47d924c30f97-kube-api-access-mr7zc\") pod \"node-resolver-q4gsb\" (UID: \"e637e650-5fc6-4adb-a97a-47d924c30f97\") " pod="openshift-dns/node-resolver-q4gsb" Nov 03 14:10:27.070598 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.070566 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6kdm\" (UniqueName: \"kubernetes.io/projected/a6466fc4-ed3d-4031-ba11-5606bd1af74b-kube-api-access-s6kdm\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:27.070695 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.070656 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfv25\" (UniqueName: \"kubernetes.io/projected/d533df4c-38b2-42b2-9cb2-b82afac33795-kube-api-access-sfv25\") pod \"aws-ebs-csi-driver-node-zkvrz\" (UID: \"d533df4c-38b2-42b2-9cb2-b82afac33795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:10:27.073365 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.073340 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7782\" (UniqueName: \"kubernetes.io/projected/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-kube-api-access-g7782\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:27.073757 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.073734 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp5h5\" (UniqueName: \"kubernetes.io/projected/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd-kube-api-access-vp5h5\") pod \"multus-additional-cni-plugins-kfkzp\" (UID: \"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd\") " pod="openshift-multus/multus-additional-cni-plugins-kfkzp" Nov 03 14:10:27.073850 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.073738 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkrl8\" (UniqueName: \"kubernetes.io/projected/f52cae01-7951-4d9e-9a57-9ba5bcdf6857-kube-api-access-fkrl8\") pod \"network-metrics-daemon-tc5nw\" (UID: \"f52cae01-7951-4d9e-9a57-9ba5bcdf6857\") " pod="openshift-multus/network-metrics-daemon-tc5nw" Nov 03 14:10:27.074538 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.074513 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgm54\" (UniqueName: \"kubernetes.io/projected/e68be0c4-d4ee-4da5-82e3-82372f811841-kube-api-access-wgm54\") pod \"node-ca-t4b94\" (UID: \"e68be0c4-d4ee-4da5-82e3-82372f811841\") " pod="openshift-image-registry/node-ca-t4b94" Nov 03 14:10:27.145248 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.145213 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:10:27.153625 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.153601 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dj8cg" Nov 03 14:10:27.157207 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:27.157160 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6466fc4_ed3d_4031_ba11_5606bd1af74b.slice/crio-3f189ffdcb2add54e66128cc837830e8011640e29d35a191fc0baabe955a96d9 WatchSource:0}: Error finding container 3f189ffdcb2add54e66128cc837830e8011640e29d35a191fc0baabe955a96d9: Status 404 returned error can't find the container with id 3f189ffdcb2add54e66128cc837830e8011640e29d35a191fc0baabe955a96d9 Nov 03 14:10:27.163422 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:27.163399 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9dee2ef_6437_4963_8e27_da5fb9c75928.slice/crio-c4519af9ec99a107edf3cad6b845dcf115f26190a900862df518f5307e681b6a WatchSource:0}: Error finding container c4519af9ec99a107edf3cad6b845dcf115f26190a900862df518f5307e681b6a: Status 404 returned error can't find the container with id c4519af9ec99a107edf3cad6b845dcf115f26190a900862df518f5307e681b6a Nov 03 14:10:27.164954 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.164896 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q4gsb" Nov 03 14:10:27.175757 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:27.175734 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode637e650_5fc6_4adb_a97a_47d924c30f97.slice/crio-dd6604a80c529a9ea0dfffe1743aecf939cd6a48243ad6ffa3691bb8cfe13087 WatchSource:0}: Error finding container dd6604a80c529a9ea0dfffe1743aecf939cd6a48243ad6ffa3691bb8cfe13087: Status 404 returned error can't find the container with id dd6604a80c529a9ea0dfffe1743aecf939cd6a48243ad6ffa3691bb8cfe13087 Nov 03 14:10:27.183248 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.183226 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t4b94" Nov 03 14:10:27.186364 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.186325 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kfkzp" Nov 03 14:10:27.194213 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:27.194191 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode68be0c4_d4ee_4da5_82e3_82372f811841.slice/crio-14d85150d7edbe1991a5ff99576317c8d72a74e40cd748fddc80df8fbf48a7c2 WatchSource:0}: Error finding container 14d85150d7edbe1991a5ff99576317c8d72a74e40cd748fddc80df8fbf48a7c2: Status 404 returned error can't find the container with id 14d85150d7edbe1991a5ff99576317c8d72a74e40cd748fddc80df8fbf48a7c2 Nov 03 14:10:27.197472 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:27.197440 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f5e798e_5caa_4d8f_a1d3_bb41a4efa4dd.slice/crio-6267a5ca51fba195deca8d7782843172823bedf31516805702604683e8d17646 WatchSource:0}: Error finding container 6267a5ca51fba195deca8d7782843172823bedf31516805702604683e8d17646: Status 404 returned error can't find the container with id 6267a5ca51fba195deca8d7782843172823bedf31516805702604683e8d17646 Nov 03 14:10:27.201531 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.201511 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-65nrx" Nov 03 14:10:27.206509 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.206489 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:27.213715 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.213694 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:10:27.215408 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.215393 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-q5cqm" Nov 03 14:10:27.218012 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:27.217986 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71c2e12f_32f9_4ab1_b74c_bc1dd28d4cce.slice/crio-2ef0fa9513f22e1f5925f997bf3f1f4eeb9577ab139104228f84d2dd73133f20 WatchSource:0}: Error finding container 2ef0fa9513f22e1f5925f997bf3f1f4eeb9577ab139104228f84d2dd73133f20: Status 404 returned error can't find the container with id 2ef0fa9513f22e1f5925f997bf3f1f4eeb9577ab139104228f84d2dd73133f20 Nov 03 14:10:27.226574 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:27.226546 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd533df4c_38b2_42b2_9cb2_b82afac33795.slice/crio-2867061ef7d1b1c91f970babfd5e957a2dc2ae00bff59a673a22086f0cf86e89 WatchSource:0}: Error finding container 2867061ef7d1b1c91f970babfd5e957a2dc2ae00bff59a673a22086f0cf86e89: Status 404 returned error can't find the container with id 2867061ef7d1b1c91f970babfd5e957a2dc2ae00bff59a673a22086f0cf86e89 Nov 03 14:10:27.227402 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:27.227380 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fe332c5_8991_4d94_9b67_c9017f4945b8.slice/crio-a7a404869ab852ee9956a1299567c7d73017b8aa057e1718974ce544148abd5f WatchSource:0}: Error finding container a7a404869ab852ee9956a1299567c7d73017b8aa057e1718974ce544148abd5f: Status 404 returned error can't find the container with id a7a404869ab852ee9956a1299567c7d73017b8aa057e1718974ce544148abd5f Nov 03 14:10:27.296609 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.296530 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 14:10:27.537680 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.537641 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef43449c-b7dc-4e84-b785-63201f80d4da-original-pull-secret\") pod \"global-pull-secret-syncer-2xrr4\" (UID: \"ef43449c-b7dc-4e84-b785-63201f80d4da\") " pod="kube-system/global-pull-secret-syncer-2xrr4" Nov 03 14:10:27.537869 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.537699 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f52cae01-7951-4d9e-9a57-9ba5bcdf6857-metrics-certs\") pod \"network-metrics-daemon-tc5nw\" (UID: \"f52cae01-7951-4d9e-9a57-9ba5bcdf6857\") " pod="openshift-multus/network-metrics-daemon-tc5nw" Nov 03 14:10:27.537869 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:27.537826 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 14:10:27.537869 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:27.537862 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 14:10:27.538007 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:27.537920 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef43449c-b7dc-4e84-b785-63201f80d4da-original-pull-secret podName:ef43449c-b7dc-4e84-b785-63201f80d4da nodeName:}" failed. No retries permitted until 2026-11-03 14:10:28.537899003 +0000 UTC m=+4.154025584 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef43449c-b7dc-4e84-b785-63201f80d4da-original-pull-secret") pod "global-pull-secret-syncer-2xrr4" (UID: "ef43449c-b7dc-4e84-b785-63201f80d4da") : object "kube-system"/"original-pull-secret" not registered Nov 03 14:10:27.538007 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:27.537944 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f52cae01-7951-4d9e-9a57-9ba5bcdf6857-metrics-certs podName:f52cae01-7951-4d9e-9a57-9ba5bcdf6857 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:28.53793319 +0000 UTC m=+4.154059769 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f52cae01-7951-4d9e-9a57-9ba5bcdf6857-metrics-certs") pod "network-metrics-daemon-tc5nw" (UID: "f52cae01-7951-4d9e-9a57-9ba5bcdf6857") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 14:10:27.638606 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.638530 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkjrd\" (UniqueName: \"kubernetes.io/projected/328dde79-0c85-47a3-b3f8-38d8cc8614c0-kube-api-access-dkjrd\") pod \"network-check-target-xkhr9\" (UID: \"328dde79-0c85-47a3-b3f8-38d8cc8614c0\") " pod="openshift-network-diagnostics/network-check-target-xkhr9" Nov 03 14:10:27.638745 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:27.638675 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 14:10:27.638745 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:27.638693 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 14:10:27.638745 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:27.638707 2577 projected.go:196] Error preparing data for projected volume kube-api-access-dkjrd for pod openshift-network-diagnostics/network-check-target-xkhr9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 14:10:27.638927 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:27.638763 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/328dde79-0c85-47a3-b3f8-38d8cc8614c0-kube-api-access-dkjrd podName:328dde79-0c85-47a3-b3f8-38d8cc8614c0 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:28.638743628 +0000 UTC m=+4.254870209 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dkjrd" (UniqueName: "kubernetes.io/projected/328dde79-0c85-47a3-b3f8-38d8cc8614c0-kube-api-access-dkjrd") pod "network-check-target-xkhr9" (UID: "328dde79-0c85-47a3-b3f8-38d8cc8614c0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 14:10:27.882459 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.882134 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 14:05:26 +0000 UTC" deadline="2028-04-07 00:24:34.738063357 +0000 UTC" Nov 03 14:10:27.882459 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.882404 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12490h14m6.855667089s" Nov 03 14:10:27.902501 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.902369 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t4b94" event={"ID":"e68be0c4-d4ee-4da5-82e3-82372f811841","Type":"ContainerStarted","Data":"14d85150d7edbe1991a5ff99576317c8d72a74e40cd748fddc80df8fbf48a7c2"} Nov 03 14:10:27.904369 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.904344 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dj8cg" event={"ID":"e9dee2ef-6437-4963-8e27-da5fb9c75928","Type":"ContainerStarted","Data":"c4519af9ec99a107edf3cad6b845dcf115f26190a900862df518f5307e681b6a"} Nov 03 14:10:27.932869 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.932819 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" event={"ID":"d533df4c-38b2-42b2-9cb2-b82afac33795","Type":"ContainerStarted","Data":"2867061ef7d1b1c91f970babfd5e957a2dc2ae00bff59a673a22086f0cf86e89"} Nov 03 14:10:27.942346 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.942304 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" event={"ID":"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce","Type":"ContainerStarted","Data":"2ef0fa9513f22e1f5925f997bf3f1f4eeb9577ab139104228f84d2dd73133f20"} Nov 03 14:10:27.946420 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.946395 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-65nrx" event={"ID":"396e7be7-0e72-4c32-a04a-22bd9bd708e1","Type":"ContainerStarted","Data":"e29b1276c9ee01b351f9edf2efe1fd2654561638304ad0b116f55d3516b832ee"} Nov 03 14:10:27.952126 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.952101 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q4gsb" event={"ID":"e637e650-5fc6-4adb-a97a-47d924c30f97","Type":"ContainerStarted","Data":"dd6604a80c529a9ea0dfffe1743aecf939cd6a48243ad6ffa3691bb8cfe13087"} Nov 03 14:10:27.960851 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.956716 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xcplz" event={"ID":"a6466fc4-ed3d-4031-ba11-5606bd1af74b","Type":"ContainerStarted","Data":"3f189ffdcb2add54e66128cc837830e8011640e29d35a191fc0baabe955a96d9"} Nov 03 14:10:27.960851 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.959533 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-171.ec2.internal" event={"ID":"3a1b8625ac3f3be5fcd1bbb72178f5c6","Type":"ContainerStarted","Data":"d03de8744420466b622bbc39eaaa70195f847bcc958bbc09ca60040ef30d51b0"} Nov 03 14:10:27.972813 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.972762 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-q5cqm" event={"ID":"3fe332c5-8991-4d94-9b67-c9017f4945b8","Type":"ContainerStarted","Data":"a7a404869ab852ee9956a1299567c7d73017b8aa057e1718974ce544148abd5f"} Nov 03 14:10:27.981379 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:27.981128 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kfkzp" event={"ID":"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd","Type":"ContainerStarted","Data":"6267a5ca51fba195deca8d7782843172823bedf31516805702604683e8d17646"} Nov 03 14:10:28.546989 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:28.546956 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef43449c-b7dc-4e84-b785-63201f80d4da-original-pull-secret\") pod \"global-pull-secret-syncer-2xrr4\" (UID: \"ef43449c-b7dc-4e84-b785-63201f80d4da\") " pod="kube-system/global-pull-secret-syncer-2xrr4" Nov 03 14:10:28.547193 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:28.547021 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f52cae01-7951-4d9e-9a57-9ba5bcdf6857-metrics-certs\") pod \"network-metrics-daemon-tc5nw\" (UID: \"f52cae01-7951-4d9e-9a57-9ba5bcdf6857\") " pod="openshift-multus/network-metrics-daemon-tc5nw" Nov 03 14:10:28.547193 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:28.547153 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 14:10:28.547363 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:28.547209 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f52cae01-7951-4d9e-9a57-9ba5bcdf6857-metrics-certs podName:f52cae01-7951-4d9e-9a57-9ba5bcdf6857 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:30.547191544 +0000 UTC m=+6.163318129 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f52cae01-7951-4d9e-9a57-9ba5bcdf6857-metrics-certs") pod "network-metrics-daemon-tc5nw" (UID: "f52cae01-7951-4d9e-9a57-9ba5bcdf6857") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 14:10:28.547646 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:28.547625 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 14:10:28.547745 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:28.547683 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef43449c-b7dc-4e84-b785-63201f80d4da-original-pull-secret podName:ef43449c-b7dc-4e84-b785-63201f80d4da nodeName:}" failed. No retries permitted until 2026-11-03 14:10:30.547666946 +0000 UTC m=+6.163793530 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef43449c-b7dc-4e84-b785-63201f80d4da-original-pull-secret") pod "global-pull-secret-syncer-2xrr4" (UID: "ef43449c-b7dc-4e84-b785-63201f80d4da") : object "kube-system"/"original-pull-secret" not registered Nov 03 14:10:28.619114 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:28.619004 2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 14:10:28.648686 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:28.648092 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkjrd\" (UniqueName: \"kubernetes.io/projected/328dde79-0c85-47a3-b3f8-38d8cc8614c0-kube-api-access-dkjrd\") pod \"network-check-target-xkhr9\" (UID: \"328dde79-0c85-47a3-b3f8-38d8cc8614c0\") " pod="openshift-network-diagnostics/network-check-target-xkhr9" Nov 03 14:10:28.648686 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:28.648276 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 14:10:28.648686 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:28.648295 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 14:10:28.648686 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:28.648307 2577 projected.go:196] Error preparing data for projected volume kube-api-access-dkjrd for pod openshift-network-diagnostics/network-check-target-xkhr9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 14:10:28.648686 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:28.648377 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/328dde79-0c85-47a3-b3f8-38d8cc8614c0-kube-api-access-dkjrd podName:328dde79-0c85-47a3-b3f8-38d8cc8614c0 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:30.648360879 +0000 UTC m=+6.264487464 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dkjrd" (UniqueName: "kubernetes.io/projected/328dde79-0c85-47a3-b3f8-38d8cc8614c0-kube-api-access-dkjrd") pod "network-check-target-xkhr9" (UID: "328dde79-0c85-47a3-b3f8-38d8cc8614c0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 14:10:28.883556 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:28.883458 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 14:05:26 +0000 UTC" deadline="2028-04-07 20:25:25.261390952 +0000 UTC" Nov 03 14:10:28.883556 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:28.883491 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12510h14m56.377903796s" Nov 03 14:10:28.888718 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:28.887944 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xrr4" Nov 03 14:10:28.888718 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:28.888055 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xrr4" podUID="ef43449c-b7dc-4e84-b785-63201f80d4da" Nov 03 14:10:28.888718 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:28.888432 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkhr9" Nov 03 14:10:28.888718 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:28.888519 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xkhr9" podUID="328dde79-0c85-47a3-b3f8-38d8cc8614c0" Nov 03 14:10:28.888718 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:28.888588 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc5nw" Nov 03 14:10:28.888718 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:28.888665 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc5nw" podUID="f52cae01-7951-4d9e-9a57-9ba5bcdf6857" Nov 03 14:10:28.988245 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:28.988202 2577 generic.go:356] "Generic (PLEG): container finished" podID="84a79a081fb234a3eba8ea0d2c88476d" containerID="1c22cb15762f0f547cf6c6ceefb39cd850e4948f17aa3fe53bc665f93e6897af" exitCode=0 Nov 03 14:10:28.988911 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:28.988886 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal" event={"ID":"84a79a081fb234a3eba8ea0d2c88476d","Type":"ContainerDied","Data":"1c22cb15762f0f547cf6c6ceefb39cd850e4948f17aa3fe53bc665f93e6897af"} Nov 03 14:10:29.034085 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:29.032992 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-171.ec2.internal" podStartSLOduration=3.032973508 podStartE2EDuration="3.032973508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 14:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:10:28.003176766 +0000 UTC m=+3.619303369" watchObservedRunningTime="2026-11-03 14:10:29.032973508 +0000 UTC m=+4.649100108" Nov 03 14:10:29.999537 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:29.999346 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal" event={"ID":"84a79a081fb234a3eba8ea0d2c88476d","Type":"ContainerStarted","Data":"c4f9f990415b112f9c4b71851c65c694ff22433c825396f5deb00e5d82cef063"} Nov 03 14:10:30.569475 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:30.568725 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef43449c-b7dc-4e84-b785-63201f80d4da-original-pull-secret\") pod \"global-pull-secret-syncer-2xrr4\" (UID: \"ef43449c-b7dc-4e84-b785-63201f80d4da\") " pod="kube-system/global-pull-secret-syncer-2xrr4" Nov 03 14:10:30.569475 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:30.568789 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f52cae01-7951-4d9e-9a57-9ba5bcdf6857-metrics-certs\") pod \"network-metrics-daemon-tc5nw\" (UID: \"f52cae01-7951-4d9e-9a57-9ba5bcdf6857\") " pod="openshift-multus/network-metrics-daemon-tc5nw" Nov 03 14:10:30.569475 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:30.568968 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 14:10:30.569475 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:30.569030 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f52cae01-7951-4d9e-9a57-9ba5bcdf6857-metrics-certs podName:f52cae01-7951-4d9e-9a57-9ba5bcdf6857 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:34.569013559 +0000 UTC m=+10.185140138 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f52cae01-7951-4d9e-9a57-9ba5bcdf6857-metrics-certs") pod "network-metrics-daemon-tc5nw" (UID: "f52cae01-7951-4d9e-9a57-9ba5bcdf6857") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 14:10:30.569475 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:30.569094 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 14:10:30.569475 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:30.569132 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef43449c-b7dc-4e84-b785-63201f80d4da-original-pull-secret podName:ef43449c-b7dc-4e84-b785-63201f80d4da nodeName:}" failed. No retries permitted until 2026-11-03 14:10:34.569120146 +0000 UTC m=+10.185246738 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef43449c-b7dc-4e84-b785-63201f80d4da-original-pull-secret") pod "global-pull-secret-syncer-2xrr4" (UID: "ef43449c-b7dc-4e84-b785-63201f80d4da") : object "kube-system"/"original-pull-secret" not registered Nov 03 14:10:30.669595 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:30.669561 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkjrd\" (UniqueName: \"kubernetes.io/projected/328dde79-0c85-47a3-b3f8-38d8cc8614c0-kube-api-access-dkjrd\") pod \"network-check-target-xkhr9\" (UID: \"328dde79-0c85-47a3-b3f8-38d8cc8614c0\") " pod="openshift-network-diagnostics/network-check-target-xkhr9" Nov 03 14:10:30.669790 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:30.669766 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 14:10:30.669871 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:30.669793 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 14:10:30.669871 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:30.669807 2577 projected.go:196] Error preparing data for projected volume kube-api-access-dkjrd for pod openshift-network-diagnostics/network-check-target-xkhr9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 14:10:30.669976 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:30.669877 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/328dde79-0c85-47a3-b3f8-38d8cc8614c0-kube-api-access-dkjrd podName:328dde79-0c85-47a3-b3f8-38d8cc8614c0 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:34.6698604 +0000 UTC m=+10.285987002 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dkjrd" (UniqueName: "kubernetes.io/projected/328dde79-0c85-47a3-b3f8-38d8cc8614c0-kube-api-access-dkjrd") pod "network-check-target-xkhr9" (UID: "328dde79-0c85-47a3-b3f8-38d8cc8614c0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 14:10:30.888790 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:30.888764 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc5nw" Nov 03 14:10:30.888930 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:30.888892 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc5nw" podUID="f52cae01-7951-4d9e-9a57-9ba5bcdf6857" Nov 03 14:10:30.889207 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:30.889186 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xrr4" Nov 03 14:10:30.889298 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:30.889271 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xrr4" podUID="ef43449c-b7dc-4e84-b785-63201f80d4da" Nov 03 14:10:30.889339 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:30.889312 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkhr9" Nov 03 14:10:30.889382 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:30.889366 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xkhr9" podUID="328dde79-0c85-47a3-b3f8-38d8cc8614c0" Nov 03 14:10:31.005342 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:31.005306 2577 generic.go:356] "Generic (PLEG): container finished" podID="1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd" containerID="27776870886ef7404c376ae0dfcfb80212a9ca04bee79ea22da0db38709d2d93" exitCode=0 Nov 03 14:10:31.005806 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:31.005393 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kfkzp" event={"ID":"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd","Type":"ContainerDied","Data":"27776870886ef7404c376ae0dfcfb80212a9ca04bee79ea22da0db38709d2d93"} Nov 03 14:10:31.011937 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:31.011908 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dj8cg" event={"ID":"e9dee2ef-6437-4963-8e27-da5fb9c75928","Type":"ContainerStarted","Data":"bc6b803ebdd945f4969a6c4037dfa6985bda85bc51bde8e8cec2101d8ef3312c"} Nov 03 14:10:31.087858 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:31.087746 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal" podStartSLOduration=5.087727565 podStartE2EDuration="5.087727565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 14:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:10:30.045390348 +0000 UTC m=+5.661516951" watchObservedRunningTime="2026-11-03 14:10:31.087727565 +0000 UTC m=+6.703854167" Nov 03 14:10:31.132749 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:31.132688 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dj8cg" podStartSLOduration=2.7183631310000003 podStartE2EDuration="6.132671302s" totalImagesPullingTime="3.414308171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:10:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:10:31.131923284 +0000 UTC m=+6.748049888" watchObservedRunningTime="2026-11-03 14:10:31.132671302 +0000 UTC m=+6.748797927" Nov 03 14:10:32.261532 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:32.261499 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dj8cg" Nov 03 14:10:32.262470 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:32.262438 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dj8cg" Nov 03 14:10:32.887075 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:32.887053 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xrr4" Nov 03 14:10:32.887185 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:32.887062 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkhr9" Nov 03 14:10:32.887185 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:32.887159 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xrr4" podUID="ef43449c-b7dc-4e84-b785-63201f80d4da" Nov 03 14:10:32.887296 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:32.887221 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xkhr9" podUID="328dde79-0c85-47a3-b3f8-38d8cc8614c0" Nov 03 14:10:32.887296 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:32.887253 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc5nw" Nov 03 14:10:32.887388 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:32.887323 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc5nw" podUID="f52cae01-7951-4d9e-9a57-9ba5bcdf6857" Nov 03 14:10:33.028511 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:33.028472 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t4b94" event={"ID":"e68be0c4-d4ee-4da5-82e3-82372f811841","Type":"ContainerStarted","Data":"06cb4308163ee18505ede78a20650000b5ee7b8824e7997c30147d049ccce261"} Nov 03 14:10:33.032238 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:33.032210 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" event={"ID":"d533df4c-38b2-42b2-9cb2-b82afac33795","Type":"ContainerStarted","Data":"52805d7f2554ea8832b812e1ce2e563b75c0a57c3c8d6360a17a2d250e8bc51d"} Nov 03 14:10:34.034883 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:34.033934 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 14:10:34.424868 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:34.424797 2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 14:10:34.605564 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:34.605468 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef43449c-b7dc-4e84-b785-63201f80d4da-original-pull-secret\") pod \"global-pull-secret-syncer-2xrr4\" (UID: \"ef43449c-b7dc-4e84-b785-63201f80d4da\") " pod="kube-system/global-pull-secret-syncer-2xrr4" Nov 03 14:10:34.605564 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:34.605553 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f52cae01-7951-4d9e-9a57-9ba5bcdf6857-metrics-certs\") pod \"network-metrics-daemon-tc5nw\" (UID: \"f52cae01-7951-4d9e-9a57-9ba5bcdf6857\") " pod="openshift-multus/network-metrics-daemon-tc5nw" Nov 03 14:10:34.605760 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:34.605684 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 14:10:34.605760 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:34.605737 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f52cae01-7951-4d9e-9a57-9ba5bcdf6857-metrics-certs podName:f52cae01-7951-4d9e-9a57-9ba5bcdf6857 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:42.605721237 +0000 UTC m=+18.221847836 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f52cae01-7951-4d9e-9a57-9ba5bcdf6857-metrics-certs") pod "network-metrics-daemon-tc5nw" (UID: "f52cae01-7951-4d9e-9a57-9ba5bcdf6857") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 14:10:34.606155 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:34.606138 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 14:10:34.606228 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:34.606186 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef43449c-b7dc-4e84-b785-63201f80d4da-original-pull-secret podName:ef43449c-b7dc-4e84-b785-63201f80d4da nodeName:}" failed. No retries permitted until 2026-11-03 14:10:42.606171696 +0000 UTC m=+18.222298289 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef43449c-b7dc-4e84-b785-63201f80d4da-original-pull-secret") pod "global-pull-secret-syncer-2xrr4" (UID: "ef43449c-b7dc-4e84-b785-63201f80d4da") : object "kube-system"/"original-pull-secret" not registered Nov 03 14:10:34.706220 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:34.706194 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkjrd\" (UniqueName: \"kubernetes.io/projected/328dde79-0c85-47a3-b3f8-38d8cc8614c0-kube-api-access-dkjrd\") pod \"network-check-target-xkhr9\" (UID: \"328dde79-0c85-47a3-b3f8-38d8cc8614c0\") " pod="openshift-network-diagnostics/network-check-target-xkhr9" Nov 03 14:10:34.706350 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:34.706309 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 14:10:34.706350 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:34.706325 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 14:10:34.706350 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:34.706337 2577 projected.go:196] Error preparing data for projected volume kube-api-access-dkjrd for pod openshift-network-diagnostics/network-check-target-xkhr9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 14:10:34.706496 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:34.706386 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/328dde79-0c85-47a3-b3f8-38d8cc8614c0-kube-api-access-dkjrd podName:328dde79-0c85-47a3-b3f8-38d8cc8614c0 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:42.706372431 +0000 UTC m=+18.322499034 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dkjrd" (UniqueName: "kubernetes.io/projected/328dde79-0c85-47a3-b3f8-38d8cc8614c0-kube-api-access-dkjrd") pod "network-check-target-xkhr9" (UID: "328dde79-0c85-47a3-b3f8-38d8cc8614c0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 14:10:34.881269 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:34.881119 2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T14:10:34.424817777Z","UUID":"0aa0529d-c86d-4cbb-b257-711bc7212138","Handler":null,"Name":"","Endpoint":""} Nov 03 14:10:34.894323 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:34.894003 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 14:10:34.894323 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:34.894042 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 14:10:34.894548 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:34.894335 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc5nw" Nov 03 14:10:34.894548 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:34.894463 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc5nw" podUID="f52cae01-7951-4d9e-9a57-9ba5bcdf6857" Nov 03 14:10:34.894655 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:34.894602 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xrr4" Nov 03 14:10:34.894705 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:34.894687 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xrr4" podUID="ef43449c-b7dc-4e84-b785-63201f80d4da" Nov 03 14:10:34.894779 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:34.894757 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkhr9" Nov 03 14:10:34.894927 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:34.894870 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xkhr9" podUID="328dde79-0c85-47a3-b3f8-38d8cc8614c0" Nov 03 14:10:35.040597 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:35.040540 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-q5cqm" event={"ID":"3fe332c5-8991-4d94-9b67-c9017f4945b8","Type":"ContainerStarted","Data":"dbae4c637e1c4de4dfce91cce16a38c7b70122810e7e95860afdd9495b83e73f"} Nov 03 14:10:35.044454 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:35.044423 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" event={"ID":"d533df4c-38b2-42b2-9cb2-b82afac33795","Type":"ContainerStarted","Data":"c0e77a69a06a989219db87a6bad65829149378181d6412e307688418c7f84330"} Nov 03 14:10:35.089619 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:35.089568 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-t4b94" podStartSLOduration=5.487163241 podStartE2EDuration="10.089550978s" totalImagesPullingTime="4.602387737s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 14:10:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:10:33.070908229 +0000 UTC m=+8.687034831" watchObservedRunningTime="2026-11-03 14:10:35.089550978 +0000 UTC m=+10.705677580" Nov 03 14:10:36.048553 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:36.048336 2577 generic.go:356] "Generic (PLEG): container finished" podID="1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd" containerID="5bdad1c802cf9cc8998f00321668b0c4cfc118fda6adf2e0ea6f2dac14495361" exitCode=0 Nov 03 14:10:36.048950 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:36.048631 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kfkzp" event={"ID":"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd","Type":"ContainerDied","Data":"5bdad1c802cf9cc8998f00321668b0c4cfc118fda6adf2e0ea6f2dac14495361"} Nov 03 14:10:36.052545 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:36.052490 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" event={"ID":"d533df4c-38b2-42b2-9cb2-b82afac33795","Type":"ContainerStarted","Data":"971bdd21dd528498a07b3af83747bccfb3aac551fa36ea5b0a1fba287620babe"} Nov 03 14:10:36.054260 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:36.054233 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-65nrx" event={"ID":"396e7be7-0e72-4c32-a04a-22bd9bd708e1","Type":"ContainerStarted","Data":"b3dc26207b9ee9d9e9d3a2fa4a503ca62574d2923858ed4d3485f4bcd5c86862"} Nov 03 14:10:36.056954 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:36.056905 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q4gsb" event={"ID":"e637e650-5fc6-4adb-a97a-47d924c30f97","Type":"ContainerStarted","Data":"a73211ca866239dd6c3845cfccc82e609bb5c133a0786c0771c6c44ebb210143"} Nov 03 14:10:36.097807 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:36.097748 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-q5cqm" podStartSLOduration=3.711037486 podStartE2EDuration="11.097736348s" totalImagesPullingTime="7.386698862s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 14:10:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:10:35.09031876 +0000 UTC m=+10.706445363" watchObservedRunningTime="2026-11-03 14:10:36.097736348 +0000 UTC m=+11.713862949" Nov 03 14:10:36.139599 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:36.139543 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-65nrx" podStartSLOduration=3.695300745 podStartE2EDuration="11.139528379s" totalImagesPullingTime="7.444227634s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 14:10:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:10:36.138897184 +0000 UTC m=+11.755023786" watchObservedRunningTime="2026-11-03 14:10:36.139528379 +0000 UTC m=+11.755654980" Nov 03 14:10:36.223594 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:36.223542 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-q4gsb" podStartSLOduration=3.49799221 podStartE2EDuration="11.223528317s" totalImagesPullingTime="7.725536107s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 14:10:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:10:36.179627734 +0000 UTC m=+11.795754336" watchObservedRunningTime="2026-11-03 14:10:36.223528317 +0000 UTC m=+11.839654901" Nov 03 14:10:36.887877 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:36.887828 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkhr9" Nov 03 14:10:36.888094 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:36.887857 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc5nw" Nov 03 14:10:36.888094 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:36.887987 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xkhr9" podUID="328dde79-0c85-47a3-b3f8-38d8cc8614c0" Nov 03 14:10:36.888094 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:36.888038 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xrr4" Nov 03 14:10:36.888252 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:36.888130 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc5nw" podUID="f52cae01-7951-4d9e-9a57-9ba5bcdf6857" Nov 03 14:10:36.888252 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:36.888202 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xrr4" podUID="ef43449c-b7dc-4e84-b785-63201f80d4da" Nov 03 14:10:38.063929 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:38.063439 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xcplz" event={"ID":"a6466fc4-ed3d-4031-ba11-5606bd1af74b","Type":"ContainerStarted","Data":"e8e166fa939bb8994d94c3e1ccc743c3b7a785042ec430a415ccd81fbdb014de"} Nov 03 14:10:38.065940 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:38.065911 2577 generic.go:356] "Generic (PLEG): container finished" podID="1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd" containerID="0dc793aba877130196fa01ab8f611e814d8d270ade124a55f7fbc073c84a6ad0" exitCode=0 Nov 03 14:10:38.066052 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:38.065959 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kfkzp" event={"ID":"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd","Type":"ContainerDied","Data":"0dc793aba877130196fa01ab8f611e814d8d270ade124a55f7fbc073c84a6ad0"} Nov 03 14:10:38.106941 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:38.106882 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" podStartSLOduration=5.255889538 podStartE2EDuration="13.106866494s" totalImagesPullingTime="7.850976956s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 14:10:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:10:36.23323587 +0000 UTC m=+11.849362472" watchObservedRunningTime="2026-11-03 14:10:38.106866494 +0000 UTC m=+13.722993096" Nov 03 14:10:38.107260 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:38.107225 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xcplz" podStartSLOduration=3.232770323 podStartE2EDuration="13.107216286s" totalImagesPullingTime="9.874445963s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 14:10:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:10:38.106976084 +0000 UTC m=+13.723102685" watchObservedRunningTime="2026-11-03 14:10:38.107216286 +0000 UTC m=+13.723342887" Nov 03 14:10:38.887272 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:38.887238 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkhr9" Nov 03 14:10:38.887446 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:38.887246 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xrr4" Nov 03 14:10:38.887446 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:38.887382 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xkhr9" podUID="328dde79-0c85-47a3-b3f8-38d8cc8614c0" Nov 03 14:10:38.887446 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:38.887406 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc5nw" Nov 03 14:10:38.887609 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:38.887490 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xrr4" podUID="ef43449c-b7dc-4e84-b785-63201f80d4da" Nov 03 14:10:38.887609 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:38.887560 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc5nw" podUID="f52cae01-7951-4d9e-9a57-9ba5bcdf6857" Nov 03 14:10:40.074328 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:40.074124 2577 generic.go:356] "Generic (PLEG): container finished" podID="1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd" containerID="0d36a983a6dc99c9069dc3bcd0b0a738e8c47b13de3419ee1eb1188f6a060ec2" exitCode=0 Nov 03 14:10:40.074328 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:40.074204 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kfkzp" event={"ID":"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd","Type":"ContainerDied","Data":"0d36a983a6dc99c9069dc3bcd0b0a738e8c47b13de3419ee1eb1188f6a060ec2"} Nov 03 14:10:40.887912 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:40.887875 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkhr9" Nov 03 14:10:40.888073 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:40.888016 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xkhr9" podUID="328dde79-0c85-47a3-b3f8-38d8cc8614c0" Nov 03 14:10:40.888073 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:40.887884 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xrr4" Nov 03 14:10:40.888193 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:40.888106 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xrr4" podUID="ef43449c-b7dc-4e84-b785-63201f80d4da" Nov 03 14:10:40.888193 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:40.887879 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc5nw" Nov 03 14:10:40.888193 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:40.888187 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc5nw" podUID="f52cae01-7951-4d9e-9a57-9ba5bcdf6857" Nov 03 14:10:42.662087 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:42.661873 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef43449c-b7dc-4e84-b785-63201f80d4da-original-pull-secret\") pod \"global-pull-secret-syncer-2xrr4\" (UID: \"ef43449c-b7dc-4e84-b785-63201f80d4da\") " pod="kube-system/global-pull-secret-syncer-2xrr4" Nov 03 14:10:42.662087 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:42.662064 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f52cae01-7951-4d9e-9a57-9ba5bcdf6857-metrics-certs\") pod \"network-metrics-daemon-tc5nw\" (UID: \"f52cae01-7951-4d9e-9a57-9ba5bcdf6857\") " pod="openshift-multus/network-metrics-daemon-tc5nw" Nov 03 14:10:42.662462 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:42.662010 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 14:10:42.662462 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:42.662165 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef43449c-b7dc-4e84-b785-63201f80d4da-original-pull-secret podName:ef43449c-b7dc-4e84-b785-63201f80d4da nodeName:}" failed. No retries permitted until 2026-11-03 14:10:58.662150004 +0000 UTC m=+34.278276594 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef43449c-b7dc-4e84-b785-63201f80d4da-original-pull-secret") pod "global-pull-secret-syncer-2xrr4" (UID: "ef43449c-b7dc-4e84-b785-63201f80d4da") : object "kube-system"/"original-pull-secret" not registered Nov 03 14:10:42.662462 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:42.662167 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 14:10:42.662462 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:42.662207 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f52cae01-7951-4d9e-9a57-9ba5bcdf6857-metrics-certs podName:f52cae01-7951-4d9e-9a57-9ba5bcdf6857 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:58.66219537 +0000 UTC m=+34.278321968 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f52cae01-7951-4d9e-9a57-9ba5bcdf6857-metrics-certs") pod "network-metrics-daemon-tc5nw" (UID: "f52cae01-7951-4d9e-9a57-9ba5bcdf6857") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 14:10:42.762877 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:42.762845 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkjrd\" (UniqueName: \"kubernetes.io/projected/328dde79-0c85-47a3-b3f8-38d8cc8614c0-kube-api-access-dkjrd\") pod \"network-check-target-xkhr9\" (UID: \"328dde79-0c85-47a3-b3f8-38d8cc8614c0\") " pod="openshift-network-diagnostics/network-check-target-xkhr9" Nov 03 14:10:42.763026 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:42.762972 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 14:10:42.763026 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:42.762989 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 14:10:42.763026 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:42.762998 2577 projected.go:196] Error preparing data for projected volume kube-api-access-dkjrd for pod openshift-network-diagnostics/network-check-target-xkhr9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 14:10:42.763117 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:42.763040 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/328dde79-0c85-47a3-b3f8-38d8cc8614c0-kube-api-access-dkjrd podName:328dde79-0c85-47a3-b3f8-38d8cc8614c0 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:58.76302672 +0000 UTC m=+34.379153312 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dkjrd" (UniqueName: "kubernetes.io/projected/328dde79-0c85-47a3-b3f8-38d8cc8614c0-kube-api-access-dkjrd") pod "network-check-target-xkhr9" (UID: "328dde79-0c85-47a3-b3f8-38d8cc8614c0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 14:10:42.887717 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:42.887681 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkhr9" Nov 03 14:10:42.887896 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:42.887721 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc5nw" Nov 03 14:10:42.887896 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:42.887693 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xrr4" Nov 03 14:10:42.887896 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:42.887806 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xkhr9" podUID="328dde79-0c85-47a3-b3f8-38d8cc8614c0" Nov 03 14:10:42.888001 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:42.887900 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc5nw" podUID="f52cae01-7951-4d9e-9a57-9ba5bcdf6857" Nov 03 14:10:42.888001 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:42.887959 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xrr4" podUID="ef43449c-b7dc-4e84-b785-63201f80d4da" Nov 03 14:10:43.082229 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:43.082195 2577 generic.go:356] "Generic (PLEG): container finished" podID="1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd" containerID="7e38f3de7779fc4b5765c2a52708d60c0655f5782980305ce78e40a72a718bbc" exitCode=0 Nov 03 14:10:43.082401 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:43.082279 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kfkzp" event={"ID":"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd","Type":"ContainerDied","Data":"7e38f3de7779fc4b5765c2a52708d60c0655f5782980305ce78e40a72a718bbc"} Nov 03 14:10:43.085066 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:43.085046 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" event={"ID":"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce","Type":"ContainerStarted","Data":"20db5e1faa673fd05285edd726ed618b9ab894f3bd61a5b97069735c1f9bd837"} Nov 03 14:10:43.085159 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:43.085119 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" event={"ID":"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce","Type":"ContainerStarted","Data":"410e851dbaf3a1e5fc27b6ac4c07b4c46d5313f551dc48477bb79daddfd008cd"} Nov 03 14:10:43.085215 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:43.085170 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" event={"ID":"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce","Type":"ContainerStarted","Data":"3209c033294f98f037ef49bb123d23518cdf3cd11714505f3dabbb8009cc5cfa"} Nov 03 14:10:43.085270 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:43.085216 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" event={"ID":"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce","Type":"ContainerStarted","Data":"b5ea59ba27f510c75590c77a3d6b7401886afa9a258485daf2c317ef2461528f"} Nov 03 14:10:43.085320 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:43.085266 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" event={"ID":"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce","Type":"ContainerStarted","Data":"bd3d6e232feff96b4814c8ac9f4c90aa5b7de82105019e7b5d7b1971f0164823"} Nov 03 14:10:43.085320 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:43.085312 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" event={"ID":"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce","Type":"ContainerStarted","Data":"95f28818199fa083792f7bbc9faa75017ca77803f64d80542e1317e89833e8c5"} Nov 03 14:10:43.996532 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:43.996496 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dj8cg" Nov 03 14:10:43.996958 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:43.996651 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 14:10:43.997115 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:43.997087 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dj8cg" Nov 03 14:10:44.089724 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:44.089693 2577 generic.go:356] "Generic (PLEG): container finished" podID="1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd" containerID="dc902db5787a87cb4794c1abafe07a682e636cc03ded505a54b6f1cd45fba344" exitCode=0 Nov 03 14:10:44.089893 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:44.089744 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kfkzp" event={"ID":"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd","Type":"ContainerDied","Data":"dc902db5787a87cb4794c1abafe07a682e636cc03ded505a54b6f1cd45fba344"} Nov 03 14:10:44.887702 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:44.887670 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xrr4" Nov 03 14:10:44.887904 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:44.887785 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc5nw" Nov 03 14:10:44.887904 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:44.887798 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xrr4" podUID="ef43449c-b7dc-4e84-b785-63201f80d4da" Nov 03 14:10:44.887904 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:44.887886 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc5nw" podUID="f52cae01-7951-4d9e-9a57-9ba5bcdf6857" Nov 03 14:10:44.888058 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:44.887945 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkhr9" Nov 03 14:10:44.888058 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:44.888012 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xkhr9" podUID="328dde79-0c85-47a3-b3f8-38d8cc8614c0" Nov 03 14:10:45.097332 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:45.097299 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kfkzp" event={"ID":"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd","Type":"ContainerStarted","Data":"d5e82ba7a5c9c24a5b53a9366044b6419698087652972dd8848c913301dcfeed"} Nov 03 14:10:45.100285 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:45.100264 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" event={"ID":"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce","Type":"ContainerStarted","Data":"9bf409ca5271a3f26faed0b18ea30729496512cd9ea2384c91281847730be686"} Nov 03 14:10:45.167532 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:45.167437 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kfkzp" podStartSLOduration=9.356863178 podStartE2EDuration="20.167424418s" totalImagesPullingTime="10.81056124s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 14:10:25 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:10:45.167146587 +0000 UTC m=+20.783273187" watchObservedRunningTime="2026-11-03 14:10:45.167424418 +0000 UTC m=+20.783551019" Nov 03 14:10:46.887742 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:46.887669 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xrr4" Nov 03 14:10:46.887742 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:46.887736 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkhr9" Nov 03 14:10:46.888399 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:46.887733 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc5nw" Nov 03 14:10:46.888399 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:46.887827 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xkhr9" podUID="328dde79-0c85-47a3-b3f8-38d8cc8614c0" Nov 03 14:10:46.888399 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:46.887886 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xrr4" podUID="ef43449c-b7dc-4e84-b785-63201f80d4da" Nov 03 14:10:46.888399 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:46.887942 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc5nw" podUID="f52cae01-7951-4d9e-9a57-9ba5bcdf6857" Nov 03 14:10:47.108147 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:47.108113 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" event={"ID":"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce","Type":"ContainerStarted","Data":"fc0589db896b5bef93f40527c234716e4573afe0792994f2a47dfdb4fd335bbe"} Nov 03 14:10:47.108427 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:47.108396 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:47.108534 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:47.108473 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:47.123746 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:47.123719 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:47.199070 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:47.198971 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" podStartSLOduration=7.3879062300000005 podStartE2EDuration="22.198958449s" totalImagesPullingTime="14.811052219s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 14:10:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:10:47.198602403 +0000 UTC m=+22.814729026" watchObservedRunningTime="2026-11-03 14:10:47.198958449 +0000 UTC m=+22.815085050" Nov 03 14:10:48.110064 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:48.110037 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:48.125156 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:48.125135 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:10:48.698658 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:48.698624 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2xrr4"] Nov 03 14:10:48.698876 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:48.698757 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xrr4" Nov 03 14:10:48.698876 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:48.698848 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xrr4" podUID="ef43449c-b7dc-4e84-b785-63201f80d4da" Nov 03 14:10:48.701373 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:48.701350 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xkhr9"] Nov 03 14:10:48.701479 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:48.701441 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkhr9" Nov 03 14:10:48.701602 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:48.701584 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xkhr9" podUID="328dde79-0c85-47a3-b3f8-38d8cc8614c0" Nov 03 14:10:48.704291 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:48.704268 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tc5nw"] Nov 03 14:10:48.704376 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:48.704353 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc5nw" Nov 03 14:10:48.704451 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:48.704434 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc5nw" podUID="f52cae01-7951-4d9e-9a57-9ba5bcdf6857" Nov 03 14:10:49.887884 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:49.887854 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc5nw" Nov 03 14:10:49.887884 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:49.887879 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xrr4" Nov 03 14:10:49.888324 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:49.887959 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkhr9" Nov 03 14:10:49.888324 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:49.887972 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xrr4" podUID="ef43449c-b7dc-4e84-b785-63201f80d4da" Nov 03 14:10:49.888324 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:49.888026 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc5nw" podUID="f52cae01-7951-4d9e-9a57-9ba5bcdf6857" Nov 03 14:10:49.888324 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:49.888109 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xkhr9" podUID="328dde79-0c85-47a3-b3f8-38d8cc8614c0" Nov 03 14:10:51.887211 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:51.887023 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xrr4" Nov 03 14:10:51.887715 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:51.887052 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc5nw" Nov 03 14:10:51.887715 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:51.887310 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xrr4" podUID="ef43449c-b7dc-4e84-b785-63201f80d4da" Nov 03 14:10:51.887715 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:51.887075 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkhr9" Nov 03 14:10:51.887715 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:51.887372 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc5nw" podUID="f52cae01-7951-4d9e-9a57-9ba5bcdf6857" Nov 03 14:10:51.887715 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:51.887459 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xkhr9" podUID="328dde79-0c85-47a3-b3f8-38d8cc8614c0" Nov 03 14:10:53.887023 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:53.886995 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xrr4" Nov 03 14:10:53.887023 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:53.887013 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc5nw" Nov 03 14:10:53.887423 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:53.887084 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xrr4" podUID="ef43449c-b7dc-4e84-b785-63201f80d4da" Nov 03 14:10:53.887423 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:53.886999 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkhr9" Nov 03 14:10:53.887423 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:53.887155 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc5nw" podUID="f52cae01-7951-4d9e-9a57-9ba5bcdf6857" Nov 03 14:10:53.887423 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:53.887205 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xkhr9" podUID="328dde79-0c85-47a3-b3f8-38d8cc8614c0" Nov 03 14:10:54.185342 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.185273 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-171.ec2.internal" event="NodeReady" Nov 03 14:10:54.185462 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.185395 2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 14:10:54.240573 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.240546 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-g28dw"] Nov 03 14:10:54.244373 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.244351 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-pkhg9"] Nov 03 14:10:54.244501 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.244485 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-g28dw" Nov 03 14:10:54.248621 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.248595 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wcb9f"] Nov 03 14:10:54.248948 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.248923 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-pkhg9" Nov 03 14:10:54.252124 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.252106 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-zgd4k"] Nov 03 14:10:54.252252 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.252234 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-wcb9f" Nov 03 14:10:54.255449 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.255432 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6c7f54c9b7-t6wbf"] Nov 03 14:10:54.255566 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.255550 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-zgd4k" Nov 03 14:10:54.260115 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.260095 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-v6pjk"] Nov 03 14:10:54.260433 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.260411 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:10:54.263363 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.263349 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-5xzwg"] Nov 03 14:10:54.263467 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.263452 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v6pjk" Nov 03 14:10:54.265913 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.265895 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 14:10:54.266334 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.266321 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q79vl"] Nov 03 14:10:54.266443 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.266429 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" Nov 03 14:10:54.269263 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.269248 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-nvj5v"] Nov 03 14:10:54.269367 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.269353 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q79vl" Nov 03 14:10:54.274252 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.274139 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-v9mxv"] Nov 03 14:10:54.275155 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.275134 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nvj5v" Nov 03 14:10:54.277699 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.277677 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-j66qt"] Nov 03 14:10:54.278249 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.277950 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-v9mxv" Nov 03 14:10:54.281464 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.281447 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-57647df665-9q52v"] Nov 03 14:10:54.281563 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.281498 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 14:10:54.281626 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.281558 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-j66qt" Nov 03 14:10:54.281717 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.281641 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-fxpd5\"" Nov 03 14:10:54.283848 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.283813 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 14:10:54.284956 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.284938 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6d7c596f48-qrcsg"] Nov 03 14:10:54.285068 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.285052 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:10:54.285972 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.285955 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 14:10:54.286067 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.286043 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 14:10:54.291176 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.291156 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-slvhl"] Nov 03 14:10:54.292829 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.292810 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:10:54.297051 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.297027 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-rjz49"] Nov 03 14:10:54.298138 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.298116 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-slvhl" Nov 03 14:10:54.300863 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.300829 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2bk97"] Nov 03 14:10:54.301103 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.300921 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rjz49" Nov 03 14:10:54.304766 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.304745 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-g28dw"] Nov 03 14:10:54.304885 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.304781 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qnjdm"] Nov 03 14:10:54.304953 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.304896 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2bk97" Nov 03 14:10:54.308287 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.308271 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-pkhg9"] Nov 03 14:10:54.308383 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.308300 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-zgd4k"] Nov 03 14:10:54.308383 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.308318 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-5xzwg"] Nov 03 14:10:54.308383 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.308377 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q79vl"] Nov 03 14:10:54.308509 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.308384 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qnjdm" Nov 03 14:10:54.308509 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.308405 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6c7f54c9b7-t6wbf"] Nov 03 14:10:54.308509 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.308424 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-v9mxv"] Nov 03 14:10:54.308653 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.308622 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-nvj5v"] Nov 03 14:10:54.308653 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.308644 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-j66qt"] Nov 03 14:10:54.308711 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.308658 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57647df665-9q52v"] Nov 03 14:10:54.308711 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.308684 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-rjz49"] Nov 03 14:10:54.308711 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.308714 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wcb9f"] Nov 03 14:10:54.313788 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.313767 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-slvhl"] Nov 03 14:10:54.327858 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.327816 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-v6pjk"] Nov 03 14:10:54.334516 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.334493 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qnjdm"] Nov 03 14:10:54.334614 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.334549 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2bk97"] Nov 03 14:10:54.334614 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.334584 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d7c596f48-qrcsg"] Nov 03 14:10:54.347728 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.347709 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/307e48f6-4a66-4c9c-a4ff-848cce9d3e0e-trusted-ca\") pod \"console-operator-7cd56b8dbc-5xzwg\" (UID: \"307e48f6-4a66-4c9c-a4ff-848cce9d3e0e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" Nov 03 14:10:54.347801 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.347738 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd3829b5-04dc-4112-b2d9-087d136ba9ec-service-ca-bundle\") pod \"insights-operator-7c4986d449-j66qt\" (UID: \"dd3829b5-04dc-4112-b2d9-087d136ba9ec\") " pod="openshift-insights/insights-operator-7c4986d449-j66qt" Nov 03 14:10:54.347801 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.347755 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/36065516-87ab-4057-b198-439305606d12-stats-auth\") pod \"router-default-6c7f54c9b7-t6wbf\" (UID: \"36065516-87ab-4057-b198-439305606d12\") " pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:10:54.347896 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.347825 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6f219713-6fd8-4580-b1c0-24e2a929c258-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zgd4k\" (UID: \"6f219713-6fd8-4580-b1c0-24e2a929c258\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zgd4k" Nov 03 14:10:54.347896 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.347861 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/307e48f6-4a66-4c9c-a4ff-848cce9d3e0e-config\") pod \"console-operator-7cd56b8dbc-5xzwg\" (UID: \"307e48f6-4a66-4c9c-a4ff-848cce9d3e0e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" Nov 03 14:10:54.347968 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.347938 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6f219713-6fd8-4580-b1c0-24e2a929c258-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zgd4k\" (UID: \"6f219713-6fd8-4580-b1c0-24e2a929c258\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zgd4k" Nov 03 14:10:54.347968 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.347955 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/307e48f6-4a66-4c9c-a4ff-848cce9d3e0e-serving-cert\") pod \"console-operator-7cd56b8dbc-5xzwg\" (UID: \"307e48f6-4a66-4c9c-a4ff-848cce9d3e0e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" Nov 03 14:10:54.348044 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.347970 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bdf183c4-a9ee-4ea8-ace1-50f307114328-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-v6pjk\" (UID: \"bdf183c4-a9ee-4ea8-ace1-50f307114328\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v6pjk" Nov 03 14:10:54.348044 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.347985 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ea7bdafa-8d46-4b28-b93b-b73352b2e269-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-q79vl\" (UID: \"ea7bdafa-8d46-4b28-b93b-b73352b2e269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q79vl" Nov 03 14:10:54.348044 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.348005 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8d74c5e4-f631-4eae-84d5-84749b53ccf8-serving-cert\") pod \"service-ca-operator-544d749c8c-nvj5v\" (UID: \"8d74c5e4-f631-4eae-84d5-84749b53ccf8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nvj5v" Nov 03 14:10:54.348180 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.348113 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrn2g\" (UniqueName: \"kubernetes.io/projected/ea7bdafa-8d46-4b28-b93b-b73352b2e269-kube-api-access-mrn2g\") pod \"cluster-samples-operator-6ddb6c8b94-q79vl\" (UID: \"ea7bdafa-8d46-4b28-b93b-b73352b2e269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q79vl" Nov 03 14:10:54.348180 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.348148 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd3829b5-04dc-4112-b2d9-087d136ba9ec-tmp\") pod \"insights-operator-7c4986d449-j66qt\" (UID: \"dd3829b5-04dc-4112-b2d9-087d136ba9ec\") " pod="openshift-insights/insights-operator-7c4986d449-j66qt" Nov 03 14:10:54.348180 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.348174 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd3829b5-04dc-4112-b2d9-087d136ba9ec-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-j66qt\" (UID: \"dd3829b5-04dc-4112-b2d9-087d136ba9ec\") " pod="openshift-insights/insights-operator-7c4986d449-j66qt" Nov 03 14:10:54.348287 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.348229 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fbfeab8a-b74e-4188-b565-b56442a01b79-ca-trust-extracted\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:10:54.348287 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.348278 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8d74c5e4-f631-4eae-84d5-84749b53ccf8-config\") pod \"service-ca-operator-544d749c8c-nvj5v\" (UID: \"8d74c5e4-f631-4eae-84d5-84749b53ccf8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nvj5v" Nov 03 14:10:54.348366 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.348312 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fbfeab8a-b74e-4188-b565-b56442a01b79-image-registry-private-configuration\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:10:54.348401 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.348367 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtr9w\" (UniqueName: \"kubernetes.io/projected/bdf183c4-a9ee-4ea8-ace1-50f307114328-kube-api-access-vtr9w\") pod \"cluster-monitoring-operator-884576c57-v6pjk\" (UID: \"bdf183c4-a9ee-4ea8-ace1-50f307114328\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v6pjk" Nov 03 14:10:54.348440 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.348415 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dd3829b5-04dc-4112-b2d9-087d136ba9ec-serving-cert\") pod \"insights-operator-7c4986d449-j66qt\" (UID: \"dd3829b5-04dc-4112-b2d9-087d136ba9ec\") " pod="openshift-insights/insights-operator-7c4986d449-j66qt" Nov 03 14:10:54.348498 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.348479 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pp5r\" (UniqueName: \"kubernetes.io/projected/00a412a1-3764-404b-9e6d-3a2bbb200781-kube-api-access-9pp5r\") pod \"kas-connection-checker-685895c7c9-g28dw\" (UID: \"00a412a1-3764-404b-9e6d-3a2bbb200781\") " pod="kube-system/kas-connection-checker-685895c7c9-g28dw" Nov 03 14:10:54.348536 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.348514 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vktv4\" (UniqueName: \"kubernetes.io/projected/4d66aab2-4671-4bbd-a4cc-b9dd9c5a766d-kube-api-access-vktv4\") pod \"kas-connection-checker-685895c7c9-wcb9f\" (UID: \"4d66aab2-4671-4bbd-a4cc-b9dd9c5a766d\") " pod="kube-system/kas-connection-checker-685895c7c9-wcb9f" Nov 03 14:10:54.348582 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.348538 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-registry-tls\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:10:54.348623 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.348579 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fbfeab8a-b74e-4188-b565-b56442a01b79-registry-certificates\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:10:54.348623 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.348614 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fbfeab8a-b74e-4188-b565-b56442a01b79-installation-pull-secrets\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:10:54.348699 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.348634 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36065516-87ab-4057-b198-439305606d12-service-ca-bundle\") pod \"router-default-6c7f54c9b7-t6wbf\" (UID: \"36065516-87ab-4057-b198-439305606d12\") " pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:10:54.348699 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.348651 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwf5m\" (UniqueName: \"kubernetes.io/projected/8d74c5e4-f631-4eae-84d5-84749b53ccf8-kube-api-access-nwf5m\") pod \"service-ca-operator-544d749c8c-nvj5v\" (UID: \"8d74c5e4-f631-4eae-84d5-84749b53ccf8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nvj5v" Nov 03 14:10:54.348765 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.348727 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fbfeab8a-b74e-4188-b565-b56442a01b79-trusted-ca\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:10:54.348765 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.348746 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k4jz\" (UniqueName: \"kubernetes.io/projected/36065516-87ab-4057-b198-439305606d12-kube-api-access-8k4jz\") pod \"router-default-6c7f54c9b7-t6wbf\" (UID: \"36065516-87ab-4057-b198-439305606d12\") " pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:10:54.348851 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.348763 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bdf183c4-a9ee-4ea8-ace1-50f307114328-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-v6pjk\" (UID: \"bdf183c4-a9ee-4ea8-ace1-50f307114328\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v6pjk" Nov 03 14:10:54.348851 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.348784 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/dd3829b5-04dc-4112-b2d9-087d136ba9ec-snapshots\") pod \"insights-operator-7c4986d449-j66qt\" (UID: \"dd3829b5-04dc-4112-b2d9-087d136ba9ec\") " pod="openshift-insights/insights-operator-7c4986d449-j66qt" Nov 03 14:10:54.348851 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.348801 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ea7bdafa-8d46-4b28-b93b-b73352b2e269-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-q79vl\" (UID: \"ea7bdafa-8d46-4b28-b93b-b73352b2e269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q79vl" Nov 03 14:10:54.348939 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.348852 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-bound-sa-token\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:10:54.348939 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.348920 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/36065516-87ab-4057-b198-439305606d12-default-certificate\") pod \"router-default-6c7f54c9b7-t6wbf\" (UID: \"36065516-87ab-4057-b198-439305606d12\") " pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:10:54.348999 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.348937 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36065516-87ab-4057-b198-439305606d12-metrics-certs\") pod \"router-default-6c7f54c9b7-t6wbf\" (UID: \"36065516-87ab-4057-b198-439305606d12\") " pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:10:54.348999 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.348954 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6m64\" (UniqueName: \"kubernetes.io/projected/dd3829b5-04dc-4112-b2d9-087d136ba9ec-kube-api-access-f6m64\") pod \"insights-operator-7c4986d449-j66qt\" (UID: \"dd3829b5-04dc-4112-b2d9-087d136ba9ec\") " pod="openshift-insights/insights-operator-7c4986d449-j66qt" Nov 03 14:10:54.348999 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.348970 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd5tr\" (UniqueName: \"kubernetes.io/projected/47c859b2-a25b-45e5-acdc-0632d472c097-kube-api-access-pd5tr\") pod \"kas-connection-checker-685895c7c9-pkhg9\" (UID: \"47c859b2-a25b-45e5-acdc-0632d472c097\") " pod="kube-system/kas-connection-checker-685895c7c9-pkhg9" Nov 03 14:10:54.348999 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.348985 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n295\" (UniqueName: \"kubernetes.io/projected/11d16990-ce0c-458f-ab1e-e9a3fed3221b-kube-api-access-8n295\") pod \"volume-data-source-validator-58c9f5f46d-v9mxv\" (UID: \"11d16990-ce0c-458f-ab1e-e9a3fed3221b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-v9mxv" Nov 03 14:10:54.349109 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.349019 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-495nb\" (UniqueName: \"kubernetes.io/projected/307e48f6-4a66-4c9c-a4ff-848cce9d3e0e-kube-api-access-495nb\") pod \"console-operator-7cd56b8dbc-5xzwg\" (UID: \"307e48f6-4a66-4c9c-a4ff-848cce9d3e0e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" Nov 03 14:10:54.349109 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.349044 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbt8t\" (UniqueName: \"kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-kube-api-access-fbt8t\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:10:54.362465 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.362442 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 14:10:54.362557 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.362467 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 14:10:54.362557 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.362515 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 14:10:54.362557 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.362525 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 14:10:54.362673 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.362644 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 14:10:54.362731 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.362679 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 14:10:54.362731 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.362708 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 14:10:54.362731 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.362724 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 14:10:54.362958 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.362943 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 14:10:54.362958 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.362950 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 14:10:54.362958 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.362949 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 14:10:54.363227 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.363210 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 14:10:54.363300 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.363241 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 14:10:54.363300 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.363263 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 14:10:54.363300 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.363277 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 14:10:54.363428 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.363302 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 14:10:54.363428 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.363314 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 14:10:54.363428 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.363335 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 14:10:54.363428 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.363344 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 14:10:54.363428 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.363341 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 14:10:54.363428 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.363338 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 14:10:54.363428 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.363364 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 14:10:54.363428 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.363372 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 14:10:54.363428 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.363372 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 14:10:54.363788 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.363727 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 14:10:54.364026 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.364012 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 14:10:54.364079 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.364023 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 14:10:54.364079 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.364036 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 14:10:54.364079 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.364053 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 14:10:54.375803 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.375787 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 14:10:54.388932 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.388913 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 14:10:54.389020 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.388970 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 14:10:54.389020 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.388999 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 14:10:54.389111 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.389095 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 14:10:54.389236 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.389215 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-nr7b5\"" Nov 03 14:10:54.389305 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.389228 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 14:10:54.389305 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.389284 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-r6xhc\"" Nov 03 14:10:54.389404 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.389309 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 14:10:54.389404 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.389350 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-jtpd7\"" Nov 03 14:10:54.389404 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.389367 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7xcd5\"" Nov 03 14:10:54.389535 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.389434 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 14:10:54.389535 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.389464 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 14:10:54.389535 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.389483 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ghsgd\"" Nov 03 14:10:54.389639 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.389609 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 14:10:54.389639 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.389624 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 14:10:54.389639 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.389630 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 14:10:54.389751 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.389630 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 14:10:54.389751 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.389633 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 14:10:54.389751 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.389654 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 14:10:54.389751 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.389673 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xpq6s\"" Nov 03 14:10:54.421777 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.421754 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dj8cg_e9dee2ef-6437-4963-8e27-da5fb9c75928/konnectivity-agent/0.log" limit=1024 Nov 03 14:10:54.449462 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.449406 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fbfeab8a-b74e-4188-b565-b56442a01b79-trusted-ca\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:10:54.449462 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.449432 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k4jz\" (UniqueName: \"kubernetes.io/projected/36065516-87ab-4057-b198-439305606d12-kube-api-access-8k4jz\") pod \"router-default-6c7f54c9b7-t6wbf\" (UID: \"36065516-87ab-4057-b198-439305606d12\") " pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:10:54.449462 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.449454 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-registry-certificates\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:10:54.449712 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.449476 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bdf183c4-a9ee-4ea8-ace1-50f307114328-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-v6pjk\" (UID: \"bdf183c4-a9ee-4ea8-ace1-50f307114328\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v6pjk" Nov 03 14:10:54.449712 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.449492 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/dd3829b5-04dc-4112-b2d9-087d136ba9ec-snapshots\") pod \"insights-operator-7c4986d449-j66qt\" (UID: \"dd3829b5-04dc-4112-b2d9-087d136ba9ec\") " pod="openshift-insights/insights-operator-7c4986d449-j66qt" Nov 03 14:10:54.449712 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.449509 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ea7bdafa-8d46-4b28-b93b-b73352b2e269-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-q79vl\" (UID: \"ea7bdafa-8d46-4b28-b93b-b73352b2e269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q79vl" Nov 03 14:10:54.449712 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.449556 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-bound-sa-token\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:10:54.449712 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.449636 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-ca-trust-extracted\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:10:54.449712 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.449666 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61060804-c519-490a-ac90-674792d50174-tmp-dir\") pod \"dns-default-2bk97\" (UID: \"61060804-c519-490a-ac90-674792d50174\") " pod="openshift-dns/dns-default-2bk97" Nov 03 14:10:54.449712 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.449695 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79cqw\" (UniqueName: \"kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-kube-api-access-79cqw\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:10:54.450061 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.449760 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/36065516-87ab-4057-b198-439305606d12-default-certificate\") pod \"router-default-6c7f54c9b7-t6wbf\" (UID: \"36065516-87ab-4057-b198-439305606d12\") " pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:10:54.450061 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.449793 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36065516-87ab-4057-b198-439305606d12-metrics-certs\") pod \"router-default-6c7f54c9b7-t6wbf\" (UID: \"36065516-87ab-4057-b198-439305606d12\") " pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:10:54.450061 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.449825 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-trusted-ca\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:10:54.450061 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.449873 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6m64\" (UniqueName: \"kubernetes.io/projected/dd3829b5-04dc-4112-b2d9-087d136ba9ec-kube-api-access-f6m64\") pod \"insights-operator-7c4986d449-j66qt\" (UID: \"dd3829b5-04dc-4112-b2d9-087d136ba9ec\") " pod="openshift-insights/insights-operator-7c4986d449-j66qt" Nov 03 14:10:54.450061 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:54.449878 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 14:10:54.450061 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.449897 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd5tr\" (UniqueName: \"kubernetes.io/projected/47c859b2-a25b-45e5-acdc-0632d472c097-kube-api-access-pd5tr\") pod \"kas-connection-checker-685895c7c9-pkhg9\" (UID: \"47c859b2-a25b-45e5-acdc-0632d472c097\") " pod="kube-system/kas-connection-checker-685895c7c9-pkhg9" Nov 03 14:10:54.450061 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.449920 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n295\" (UniqueName: \"kubernetes.io/projected/11d16990-ce0c-458f-ab1e-e9a3fed3221b-kube-api-access-8n295\") pod \"volume-data-source-validator-58c9f5f46d-v9mxv\" (UID: \"11d16990-ce0c-458f-ab1e-e9a3fed3221b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-v9mxv" Nov 03 14:10:54.450061 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:54.449942 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdf183c4-a9ee-4ea8-ace1-50f307114328-cluster-monitoring-operator-tls podName:bdf183c4-a9ee-4ea8-ace1-50f307114328 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:54.949924094 +0000 UTC m=+30.566050674 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bdf183c4-a9ee-4ea8-ace1-50f307114328-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-v6pjk" (UID: "bdf183c4-a9ee-4ea8-ace1-50f307114328") : secret "cluster-monitoring-operator-tls" not found Nov 03 14:10:54.450061 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.449971 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-495nb\" (UniqueName: \"kubernetes.io/projected/307e48f6-4a66-4c9c-a4ff-848cce9d3e0e-kube-api-access-495nb\") pod \"console-operator-7cd56b8dbc-5xzwg\" (UID: \"307e48f6-4a66-4c9c-a4ff-848cce9d3e0e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" Nov 03 14:10:54.450061 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.449998 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-installation-pull-secrets\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:10:54.450061 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.450023 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbt8t\" (UniqueName: \"kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-kube-api-access-fbt8t\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:10:54.450061 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.450048 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-626zm\" (UniqueName: \"kubernetes.io/projected/840ddc67-db99-4d3f-ba2d-51632ac3912c-kube-api-access-626zm\") pod \"network-check-source-6f78d59f99-rjz49\" (UID: \"840ddc67-db99-4d3f-ba2d-51632ac3912c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rjz49" Nov 03 14:10:54.450628 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.450077 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4w2t\" (UniqueName: \"kubernetes.io/projected/61060804-c519-490a-ac90-674792d50174-kube-api-access-h4w2t\") pod \"dns-default-2bk97\" (UID: \"61060804-c519-490a-ac90-674792d50174\") " pod="openshift-dns/dns-default-2bk97" Nov 03 14:10:54.450628 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.450105 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/307e48f6-4a66-4c9c-a4ff-848cce9d3e0e-trusted-ca\") pod \"console-operator-7cd56b8dbc-5xzwg\" (UID: \"307e48f6-4a66-4c9c-a4ff-848cce9d3e0e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" Nov 03 14:10:54.450628 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.450140 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd3829b5-04dc-4112-b2d9-087d136ba9ec-service-ca-bundle\") pod \"insights-operator-7c4986d449-j66qt\" (UID: \"dd3829b5-04dc-4112-b2d9-087d136ba9ec\") " pod="openshift-insights/insights-operator-7c4986d449-j66qt" Nov 03 14:10:54.450628 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.450165 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/36065516-87ab-4057-b198-439305606d12-stats-auth\") pod \"router-default-6c7f54c9b7-t6wbf\" (UID: \"36065516-87ab-4057-b198-439305606d12\") " pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:10:54.450628 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:54.450179 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 14:10:54.450628 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.450191 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-image-registry-private-configuration\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:10:54.450628 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:54.450231 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36065516-87ab-4057-b198-439305606d12-metrics-certs podName:36065516-87ab-4057-b198-439305606d12 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:54.950212898 +0000 UTC m=+30.566339498 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/36065516-87ab-4057-b198-439305606d12-metrics-certs") pod "router-default-6c7f54c9b7-t6wbf" (UID: "36065516-87ab-4057-b198-439305606d12") : secret "router-metrics-certs-default" not found Nov 03 14:10:54.450628 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.450252 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/61060804-c519-490a-ac90-674792d50174-config-volume\") pod \"dns-default-2bk97\" (UID: \"61060804-c519-490a-ac90-674792d50174\") " pod="openshift-dns/dns-default-2bk97" Nov 03 14:10:54.450628 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.450280 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-registry-tls\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:10:54.450628 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.450372 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/dd3829b5-04dc-4112-b2d9-087d136ba9ec-snapshots\") pod \"insights-operator-7c4986d449-j66qt\" (UID: \"dd3829b5-04dc-4112-b2d9-087d136ba9ec\") " pod="openshift-insights/insights-operator-7c4986d449-j66qt" Nov 03 14:10:54.450628 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.450388 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bf085d57-4ae4-4add-9402-9ee1ce5d795a-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-slvhl\" (UID: \"bf085d57-4ae4-4add-9402-9ee1ce5d795a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-slvhl" Nov 03 14:10:54.450628 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.450505 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ea7bdafa-8d46-4b28-b93b-b73352b2e269-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-q79vl\" (UID: \"ea7bdafa-8d46-4b28-b93b-b73352b2e269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q79vl" Nov 03 14:10:54.450628 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.450551 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6f219713-6fd8-4580-b1c0-24e2a929c258-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zgd4k\" (UID: \"6f219713-6fd8-4580-b1c0-24e2a929c258\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zgd4k" Nov 03 14:10:54.451215 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:54.450640 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 14:10:54.451215 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.450645 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fbfeab8a-b74e-4188-b565-b56442a01b79-trusted-ca\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:10:54.451215 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:54.450692 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f219713-6fd8-4580-b1c0-24e2a929c258-networking-console-plugin-cert podName:6f219713-6fd8-4580-b1c0-24e2a929c258 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:54.95067638 +0000 UTC m=+30.566802973 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6f219713-6fd8-4580-b1c0-24e2a929c258-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zgd4k" (UID: "6f219713-6fd8-4580-b1c0-24e2a929c258") : secret "networking-console-plugin-cert" not found Nov 03 14:10:54.451215 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.450712 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/307e48f6-4a66-4c9c-a4ff-848cce9d3e0e-config\") pod \"console-operator-7cd56b8dbc-5xzwg\" (UID: \"307e48f6-4a66-4c9c-a4ff-848cce9d3e0e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" Nov 03 14:10:54.451215 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.450844 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-bound-sa-token\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:10:54.451215 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.450958 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd3829b5-04dc-4112-b2d9-087d136ba9ec-service-ca-bundle\") pod \"insights-operator-7c4986d449-j66qt\" (UID: \"dd3829b5-04dc-4112-b2d9-087d136ba9ec\") " pod="openshift-insights/insights-operator-7c4986d449-j66qt" Nov 03 14:10:54.451215 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.450962 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrqb5\" (UniqueName: \"kubernetes.io/projected/63bfb9d0-9e87-44c6-a6f7-2f7b992e20f6-kube-api-access-hrqb5\") pod \"ingress-canary-qnjdm\" (UID: \"63bfb9d0-9e87-44c6-a6f7-2f7b992e20f6\") " pod="openshift-ingress-canary/ingress-canary-qnjdm" Nov 03 14:10:54.451215 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.451033 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6f219713-6fd8-4580-b1c0-24e2a929c258-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zgd4k\" (UID: \"6f219713-6fd8-4580-b1c0-24e2a929c258\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zgd4k" Nov 03 14:10:54.451215 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.451061 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/307e48f6-4a66-4c9c-a4ff-848cce9d3e0e-serving-cert\") pod \"console-operator-7cd56b8dbc-5xzwg\" (UID: \"307e48f6-4a66-4c9c-a4ff-848cce9d3e0e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" Nov 03 14:10:54.451215 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.451091 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61060804-c519-490a-ac90-674792d50174-metrics-tls\") pod \"dns-default-2bk97\" (UID: \"61060804-c519-490a-ac90-674792d50174\") " pod="openshift-dns/dns-default-2bk97" Nov 03 14:10:54.451699 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.451321 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/307e48f6-4a66-4c9c-a4ff-848cce9d3e0e-trusted-ca\") pod \"console-operator-7cd56b8dbc-5xzwg\" (UID: \"307e48f6-4a66-4c9c-a4ff-848cce9d3e0e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" Nov 03 14:10:54.451699 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.451342 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/307e48f6-4a66-4c9c-a4ff-848cce9d3e0e-config\") pod \"console-operator-7cd56b8dbc-5xzwg\" (UID: \"307e48f6-4a66-4c9c-a4ff-848cce9d3e0e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" Nov 03 14:10:54.451699 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.451420 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bdf183c4-a9ee-4ea8-ace1-50f307114328-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-v6pjk\" (UID: \"bdf183c4-a9ee-4ea8-ace1-50f307114328\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v6pjk" Nov 03 14:10:54.451699 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.451473 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ea7bdafa-8d46-4b28-b93b-b73352b2e269-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-q79vl\" (UID: \"ea7bdafa-8d46-4b28-b93b-b73352b2e269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q79vl" Nov 03 14:10:54.451699 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.451517 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8d74c5e4-f631-4eae-84d5-84749b53ccf8-serving-cert\") pod \"service-ca-operator-544d749c8c-nvj5v\" (UID: \"8d74c5e4-f631-4eae-84d5-84749b53ccf8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nvj5v" Nov 03 14:10:54.451699 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.451547 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbw2q\" (UniqueName: \"kubernetes.io/projected/bf085d57-4ae4-4add-9402-9ee1ce5d795a-kube-api-access-pbw2q\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-slvhl\" (UID: \"bf085d57-4ae4-4add-9402-9ee1ce5d795a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-slvhl" Nov 03 14:10:54.451699 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.451583 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrn2g\" (UniqueName: \"kubernetes.io/projected/ea7bdafa-8d46-4b28-b93b-b73352b2e269-kube-api-access-mrn2g\") pod \"cluster-samples-operator-6ddb6c8b94-q79vl\" (UID: \"ea7bdafa-8d46-4b28-b93b-b73352b2e269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q79vl" Nov 03 14:10:54.451699 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:54.451598 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 14:10:54.451699 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:54.451654 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea7bdafa-8d46-4b28-b93b-b73352b2e269-samples-operator-tls podName:ea7bdafa-8d46-4b28-b93b-b73352b2e269 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:54.951640672 +0000 UTC m=+30.567767257 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ea7bdafa-8d46-4b28-b93b-b73352b2e269-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-q79vl" (UID: "ea7bdafa-8d46-4b28-b93b-b73352b2e269") : secret "samples-operator-tls" not found Nov 03 14:10:54.451699 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.451608 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd3829b5-04dc-4112-b2d9-087d136ba9ec-tmp\") pod \"insights-operator-7c4986d449-j66qt\" (UID: \"dd3829b5-04dc-4112-b2d9-087d136ba9ec\") " pod="openshift-insights/insights-operator-7c4986d449-j66qt" Nov 03 14:10:54.451699 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.451694 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd3829b5-04dc-4112-b2d9-087d136ba9ec-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-j66qt\" (UID: \"dd3829b5-04dc-4112-b2d9-087d136ba9ec\") " pod="openshift-insights/insights-operator-7c4986d449-j66qt" Nov 03 14:10:54.452271 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.451725 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fbfeab8a-b74e-4188-b565-b56442a01b79-ca-trust-extracted\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:10:54.452271 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.451753 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8d74c5e4-f631-4eae-84d5-84749b53ccf8-config\") pod \"service-ca-operator-544d749c8c-nvj5v\" (UID: \"8d74c5e4-f631-4eae-84d5-84749b53ccf8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nvj5v" Nov 03 14:10:54.452271 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.451781 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fbfeab8a-b74e-4188-b565-b56442a01b79-image-registry-private-configuration\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:10:54.452271 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.451809 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtr9w\" (UniqueName: \"kubernetes.io/projected/bdf183c4-a9ee-4ea8-ace1-50f307114328-kube-api-access-vtr9w\") pod \"cluster-monitoring-operator-884576c57-v6pjk\" (UID: \"bdf183c4-a9ee-4ea8-ace1-50f307114328\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v6pjk" Nov 03 14:10:54.452271 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.451853 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dd3829b5-04dc-4112-b2d9-087d136ba9ec-serving-cert\") pod \"insights-operator-7c4986d449-j66qt\" (UID: \"dd3829b5-04dc-4112-b2d9-087d136ba9ec\") " pod="openshift-insights/insights-operator-7c4986d449-j66qt" Nov 03 14:10:54.452271 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.451880 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63bfb9d0-9e87-44c6-a6f7-2f7b992e20f6-cert\") pod \"ingress-canary-qnjdm\" (UID: \"63bfb9d0-9e87-44c6-a6f7-2f7b992e20f6\") " pod="openshift-ingress-canary/ingress-canary-qnjdm" Nov 03 14:10:54.452271 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.451896 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd3829b5-04dc-4112-b2d9-087d136ba9ec-tmp\") pod \"insights-operator-7c4986d449-j66qt\" (UID: \"dd3829b5-04dc-4112-b2d9-087d136ba9ec\") " pod="openshift-insights/insights-operator-7c4986d449-j66qt" Nov 03 14:10:54.452271 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.451909 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6f219713-6fd8-4580-b1c0-24e2a929c258-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zgd4k\" (UID: \"6f219713-6fd8-4580-b1c0-24e2a929c258\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zgd4k" Nov 03 14:10:54.452271 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.451911 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pp5r\" (UniqueName: \"kubernetes.io/projected/00a412a1-3764-404b-9e6d-3a2bbb200781-kube-api-access-9pp5r\") pod \"kas-connection-checker-685895c7c9-g28dw\" (UID: \"00a412a1-3764-404b-9e6d-3a2bbb200781\") " pod="kube-system/kas-connection-checker-685895c7c9-g28dw" Nov 03 14:10:54.452271 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.451962 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vktv4\" (UniqueName: \"kubernetes.io/projected/4d66aab2-4671-4bbd-a4cc-b9dd9c5a766d-kube-api-access-vktv4\") pod \"kas-connection-checker-685895c7c9-wcb9f\" (UID: \"4d66aab2-4671-4bbd-a4cc-b9dd9c5a766d\") " pod="kube-system/kas-connection-checker-685895c7c9-wcb9f" Nov 03 14:10:54.452271 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.451992 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bf085d57-4ae4-4add-9402-9ee1ce5d795a-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-slvhl\" (UID: \"bf085d57-4ae4-4add-9402-9ee1ce5d795a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-slvhl" Nov 03 14:10:54.452271 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.452029 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-registry-tls\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:10:54.452271 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.452066 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fbfeab8a-b74e-4188-b565-b56442a01b79-registry-certificates\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:10:54.452271 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.452090 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fbfeab8a-b74e-4188-b565-b56442a01b79-installation-pull-secrets\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:10:54.452271 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.452121 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36065516-87ab-4057-b198-439305606d12-service-ca-bundle\") pod \"router-default-6c7f54c9b7-t6wbf\" (UID: \"36065516-87ab-4057-b198-439305606d12\") " pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:10:54.452271 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.452143 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bdf183c4-a9ee-4ea8-ace1-50f307114328-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-v6pjk\" (UID: \"bdf183c4-a9ee-4ea8-ace1-50f307114328\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v6pjk" Nov 03 14:10:54.453044 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.452155 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwf5m\" (UniqueName: \"kubernetes.io/projected/8d74c5e4-f631-4eae-84d5-84749b53ccf8-kube-api-access-nwf5m\") pod \"service-ca-operator-544d749c8c-nvj5v\" (UID: \"8d74c5e4-f631-4eae-84d5-84749b53ccf8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nvj5v" Nov 03 14:10:54.453044 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.452439 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fbfeab8a-b74e-4188-b565-b56442a01b79-ca-trust-extracted\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:10:54.453044 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.452717 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd3829b5-04dc-4112-b2d9-087d136ba9ec-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-j66qt\" (UID: \"dd3829b5-04dc-4112-b2d9-087d136ba9ec\") " pod="openshift-insights/insights-operator-7c4986d449-j66qt" Nov 03 14:10:54.453044 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.452948 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8d74c5e4-f631-4eae-84d5-84749b53ccf8-config\") pod \"service-ca-operator-544d749c8c-nvj5v\" (UID: \"8d74c5e4-f631-4eae-84d5-84749b53ccf8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nvj5v" Nov 03 14:10:54.453228 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:54.453070 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 14:10:54.453228 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:54.453089 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57647df665-9q52v: secret "image-registry-tls" not found Nov 03 14:10:54.453566 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.453544 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fbfeab8a-b74e-4188-b565-b56442a01b79-registry-certificates\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:10:54.454264 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:54.454009 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/36065516-87ab-4057-b198-439305606d12-service-ca-bundle podName:36065516-87ab-4057-b198-439305606d12 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:54.953991292 +0000 UTC m=+30.570117893 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/36065516-87ab-4057-b198-439305606d12-service-ca-bundle") pod "router-default-6c7f54c9b7-t6wbf" (UID: "36065516-87ab-4057-b198-439305606d12") : configmap references non-existent config key: service-ca.crt Nov 03 14:10:54.454264 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.454200 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/307e48f6-4a66-4c9c-a4ff-848cce9d3e0e-serving-cert\") pod \"console-operator-7cd56b8dbc-5xzwg\" (UID: \"307e48f6-4a66-4c9c-a4ff-848cce9d3e0e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" Nov 03 14:10:54.454417 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.454282 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8d74c5e4-f631-4eae-84d5-84749b53ccf8-serving-cert\") pod \"service-ca-operator-544d749c8c-nvj5v\" (UID: \"8d74c5e4-f631-4eae-84d5-84749b53ccf8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nvj5v" Nov 03 14:10:54.454476 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.454414 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/36065516-87ab-4057-b198-439305606d12-stats-auth\") pod \"router-default-6c7f54c9b7-t6wbf\" (UID: \"36065516-87ab-4057-b198-439305606d12\") " pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:10:54.454524 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:54.454510 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-registry-tls podName:fbfeab8a-b74e-4188-b565-b56442a01b79 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:54.954493722 +0000 UTC m=+30.570620309 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-registry-tls") pod "image-registry-57647df665-9q52v" (UID: "fbfeab8a-b74e-4188-b565-b56442a01b79") : secret "image-registry-tls" not found Nov 03 14:10:54.454727 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.454707 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/36065516-87ab-4057-b198-439305606d12-default-certificate\") pod \"router-default-6c7f54c9b7-t6wbf\" (UID: \"36065516-87ab-4057-b198-439305606d12\") " pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:10:54.454895 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.454878 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fbfeab8a-b74e-4188-b565-b56442a01b79-image-registry-private-configuration\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:10:54.455491 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.455476 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dd3829b5-04dc-4112-b2d9-087d136ba9ec-serving-cert\") pod \"insights-operator-7c4986d449-j66qt\" (UID: \"dd3829b5-04dc-4112-b2d9-087d136ba9ec\") " pod="openshift-insights/insights-operator-7c4986d449-j66qt" Nov 03 14:10:54.455883 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.455867 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fbfeab8a-b74e-4188-b565-b56442a01b79-installation-pull-secrets\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:10:54.471750 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.471726 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-bound-sa-token\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:10:54.472712 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.472692 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-495nb\" (UniqueName: \"kubernetes.io/projected/307e48f6-4a66-4c9c-a4ff-848cce9d3e0e-kube-api-access-495nb\") pod \"console-operator-7cd56b8dbc-5xzwg\" (UID: \"307e48f6-4a66-4c9c-a4ff-848cce9d3e0e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" Nov 03 14:10:54.473315 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.473295 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k4jz\" (UniqueName: \"kubernetes.io/projected/36065516-87ab-4057-b198-439305606d12-kube-api-access-8k4jz\") pod \"router-default-6c7f54c9b7-t6wbf\" (UID: \"36065516-87ab-4057-b198-439305606d12\") " pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:10:54.473408 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.473388 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd5tr\" (UniqueName: \"kubernetes.io/projected/47c859b2-a25b-45e5-acdc-0632d472c097-kube-api-access-pd5tr\") pod \"kas-connection-checker-685895c7c9-pkhg9\" (UID: \"47c859b2-a25b-45e5-acdc-0632d472c097\") " pod="kube-system/kas-connection-checker-685895c7c9-pkhg9" Nov 03 14:10:54.473447 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.473436 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n295\" (UniqueName: \"kubernetes.io/projected/11d16990-ce0c-458f-ab1e-e9a3fed3221b-kube-api-access-8n295\") pod \"volume-data-source-validator-58c9f5f46d-v9mxv\" (UID: \"11d16990-ce0c-458f-ab1e-e9a3fed3221b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-v9mxv" Nov 03 14:10:54.473490 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.473471 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbt8t\" (UniqueName: \"kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-kube-api-access-fbt8t\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:10:54.473929 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.473905 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6m64\" (UniqueName: \"kubernetes.io/projected/dd3829b5-04dc-4112-b2d9-087d136ba9ec-kube-api-access-f6m64\") pod \"insights-operator-7c4986d449-j66qt\" (UID: \"dd3829b5-04dc-4112-b2d9-087d136ba9ec\") " pod="openshift-insights/insights-operator-7c4986d449-j66qt" Nov 03 14:10:54.474677 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.474651 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtr9w\" (UniqueName: \"kubernetes.io/projected/bdf183c4-a9ee-4ea8-ace1-50f307114328-kube-api-access-vtr9w\") pod \"cluster-monitoring-operator-884576c57-v6pjk\" (UID: \"bdf183c4-a9ee-4ea8-ace1-50f307114328\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v6pjk" Nov 03 14:10:54.475805 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.475780 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pp5r\" (UniqueName: \"kubernetes.io/projected/00a412a1-3764-404b-9e6d-3a2bbb200781-kube-api-access-9pp5r\") pod \"kas-connection-checker-685895c7c9-g28dw\" (UID: \"00a412a1-3764-404b-9e6d-3a2bbb200781\") " pod="kube-system/kas-connection-checker-685895c7c9-g28dw" Nov 03 14:10:54.476055 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.476036 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrn2g\" (UniqueName: \"kubernetes.io/projected/ea7bdafa-8d46-4b28-b93b-b73352b2e269-kube-api-access-mrn2g\") pod \"cluster-samples-operator-6ddb6c8b94-q79vl\" (UID: \"ea7bdafa-8d46-4b28-b93b-b73352b2e269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q79vl" Nov 03 14:10:54.476567 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.476548 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwf5m\" (UniqueName: \"kubernetes.io/projected/8d74c5e4-f631-4eae-84d5-84749b53ccf8-kube-api-access-nwf5m\") pod \"service-ca-operator-544d749c8c-nvj5v\" (UID: \"8d74c5e4-f631-4eae-84d5-84749b53ccf8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nvj5v" Nov 03 14:10:54.477356 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.477334 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vktv4\" (UniqueName: \"kubernetes.io/projected/4d66aab2-4671-4bbd-a4cc-b9dd9c5a766d-kube-api-access-vktv4\") pod \"kas-connection-checker-685895c7c9-wcb9f\" (UID: \"4d66aab2-4671-4bbd-a4cc-b9dd9c5a766d\") " pod="kube-system/kas-connection-checker-685895c7c9-wcb9f" Nov 03 14:10:54.553463 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.553438 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63bfb9d0-9e87-44c6-a6f7-2f7b992e20f6-cert\") pod \"ingress-canary-qnjdm\" (UID: \"63bfb9d0-9e87-44c6-a6f7-2f7b992e20f6\") " pod="openshift-ingress-canary/ingress-canary-qnjdm" Nov 03 14:10:54.553463 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.553470 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bf085d57-4ae4-4add-9402-9ee1ce5d795a-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-slvhl\" (UID: \"bf085d57-4ae4-4add-9402-9ee1ce5d795a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-slvhl" Nov 03 14:10:54.553661 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.553610 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-registry-certificates\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:10:54.553717 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.553669 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-ca-trust-extracted\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:10:54.553873 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.553850 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61060804-c519-490a-ac90-674792d50174-tmp-dir\") pod \"dns-default-2bk97\" (UID: \"61060804-c519-490a-ac90-674792d50174\") " pod="openshift-dns/dns-default-2bk97" Nov 03 14:10:54.553977 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.553905 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79cqw\" (UniqueName: \"kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-kube-api-access-79cqw\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:10:54.553977 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.553948 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-trusted-ca\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:10:54.554072 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.553980 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-installation-pull-secrets\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:10:54.554072 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.554009 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-626zm\" (UniqueName: \"kubernetes.io/projected/840ddc67-db99-4d3f-ba2d-51632ac3912c-kube-api-access-626zm\") pod \"network-check-source-6f78d59f99-rjz49\" (UID: \"840ddc67-db99-4d3f-ba2d-51632ac3912c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rjz49" Nov 03 14:10:54.554072 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.554018 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bf085d57-4ae4-4add-9402-9ee1ce5d795a-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-slvhl\" (UID: \"bf085d57-4ae4-4add-9402-9ee1ce5d795a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-slvhl" Nov 03 14:10:54.554072 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.554036 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4w2t\" (UniqueName: \"kubernetes.io/projected/61060804-c519-490a-ac90-674792d50174-kube-api-access-h4w2t\") pod \"dns-default-2bk97\" (UID: \"61060804-c519-490a-ac90-674792d50174\") " pod="openshift-dns/dns-default-2bk97" Nov 03 14:10:54.554271 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.554073 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-image-registry-private-configuration\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:10:54.554271 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.554101 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/61060804-c519-490a-ac90-674792d50174-config-volume\") pod \"dns-default-2bk97\" (UID: \"61060804-c519-490a-ac90-674792d50174\") " pod="openshift-dns/dns-default-2bk97" Nov 03 14:10:54.554271 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.554125 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-registry-tls\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:10:54.554271 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.554147 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bf085d57-4ae4-4add-9402-9ee1ce5d795a-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-slvhl\" (UID: \"bf085d57-4ae4-4add-9402-9ee1ce5d795a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-slvhl" Nov 03 14:10:54.554271 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.554165 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-ca-trust-extracted\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:10:54.554271 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.554191 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-bound-sa-token\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:10:54.554271 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.554209 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-registry-certificates\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:10:54.554271 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.554255 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrqb5\" (UniqueName: \"kubernetes.io/projected/63bfb9d0-9e87-44c6-a6f7-2f7b992e20f6-kube-api-access-hrqb5\") pod \"ingress-canary-qnjdm\" (UID: \"63bfb9d0-9e87-44c6-a6f7-2f7b992e20f6\") " pod="openshift-ingress-canary/ingress-canary-qnjdm" Nov 03 14:10:54.554271 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.554266 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61060804-c519-490a-ac90-674792d50174-tmp-dir\") pod \"dns-default-2bk97\" (UID: \"61060804-c519-490a-ac90-674792d50174\") " pod="openshift-dns/dns-default-2bk97" Nov 03 14:10:54.554728 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.554300 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61060804-c519-490a-ac90-674792d50174-metrics-tls\") pod \"dns-default-2bk97\" (UID: \"61060804-c519-490a-ac90-674792d50174\") " pod="openshift-dns/dns-default-2bk97" Nov 03 14:10:54.554728 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.554343 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbw2q\" (UniqueName: \"kubernetes.io/projected/bf085d57-4ae4-4add-9402-9ee1ce5d795a-kube-api-access-pbw2q\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-slvhl\" (UID: \"bf085d57-4ae4-4add-9402-9ee1ce5d795a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-slvhl" Nov 03 14:10:54.554728 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:54.554444 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 14:10:54.554728 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:54.554458 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d7c596f48-qrcsg: secret "image-registry-tls" not found Nov 03 14:10:54.554728 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:54.554504 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-registry-tls podName:b3260ef2-8ff4-4b70-a3a0-e5386a27d210 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:55.054488837 +0000 UTC m=+30.670615443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-registry-tls") pod "image-registry-6d7c596f48-qrcsg" (UID: "b3260ef2-8ff4-4b70-a3a0-e5386a27d210") : secret "image-registry-tls" not found Nov 03 14:10:54.554728 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:54.554552 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 14:10:54.554728 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:54.554589 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61060804-c519-490a-ac90-674792d50174-metrics-tls podName:61060804-c519-490a-ac90-674792d50174 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:55.054578979 +0000 UTC m=+30.670705572 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61060804-c519-490a-ac90-674792d50174-metrics-tls") pod "dns-default-2bk97" (UID: "61060804-c519-490a-ac90-674792d50174") : secret "dns-default-metrics-tls" not found Nov 03 14:10:54.555135 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.554997 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-trusted-ca\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:10:54.555135 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.555014 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/61060804-c519-490a-ac90-674792d50174-config-volume\") pod \"dns-default-2bk97\" (UID: \"61060804-c519-490a-ac90-674792d50174\") " pod="openshift-dns/dns-default-2bk97" Nov 03 14:10:54.556090 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.556063 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63bfb9d0-9e87-44c6-a6f7-2f7b992e20f6-cert\") pod \"ingress-canary-qnjdm\" (UID: \"63bfb9d0-9e87-44c6-a6f7-2f7b992e20f6\") " pod="openshift-ingress-canary/ingress-canary-qnjdm" Nov 03 14:10:54.556631 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.556609 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-installation-pull-secrets\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:10:54.556918 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.556898 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-image-registry-private-configuration\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:10:54.556980 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.556953 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bf085d57-4ae4-4add-9402-9ee1ce5d795a-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-slvhl\" (UID: \"bf085d57-4ae4-4add-9402-9ee1ce5d795a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-slvhl" Nov 03 14:10:54.559366 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.559351 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-g28dw" Nov 03 14:10:54.568190 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.568169 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-pkhg9" Nov 03 14:10:54.575950 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.575929 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-wcb9f" Nov 03 14:10:54.577208 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.577182 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-bound-sa-token\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:10:54.577423 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.577401 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrqb5\" (UniqueName: \"kubernetes.io/projected/63bfb9d0-9e87-44c6-a6f7-2f7b992e20f6-kube-api-access-hrqb5\") pod \"ingress-canary-qnjdm\" (UID: \"63bfb9d0-9e87-44c6-a6f7-2f7b992e20f6\") " pod="openshift-ingress-canary/ingress-canary-qnjdm" Nov 03 14:10:54.577557 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.577536 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-626zm\" (UniqueName: \"kubernetes.io/projected/840ddc67-db99-4d3f-ba2d-51632ac3912c-kube-api-access-626zm\") pod \"network-check-source-6f78d59f99-rjz49\" (UID: \"840ddc67-db99-4d3f-ba2d-51632ac3912c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rjz49" Nov 03 14:10:54.577625 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.577596 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4w2t\" (UniqueName: \"kubernetes.io/projected/61060804-c519-490a-ac90-674792d50174-kube-api-access-h4w2t\") pod \"dns-default-2bk97\" (UID: \"61060804-c519-490a-ac90-674792d50174\") " pod="openshift-dns/dns-default-2bk97" Nov 03 14:10:54.577684 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.577672 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbw2q\" (UniqueName: \"kubernetes.io/projected/bf085d57-4ae4-4add-9402-9ee1ce5d795a-kube-api-access-pbw2q\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-slvhl\" (UID: \"bf085d57-4ae4-4add-9402-9ee1ce5d795a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-slvhl" Nov 03 14:10:54.578701 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.578685 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79cqw\" (UniqueName: \"kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-kube-api-access-79cqw\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:10:54.610299 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.610265 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" Nov 03 14:10:54.624618 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.623903 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nvj5v" Nov 03 14:10:54.630251 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.630228 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-v9mxv" Nov 03 14:10:54.642204 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.641933 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-j66qt" Nov 03 14:10:54.662030 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.661788 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-slvhl" Nov 03 14:10:54.669508 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.669442 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rjz49" Nov 03 14:10:54.677813 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.677624 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qnjdm" Nov 03 14:10:54.705867 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.705787 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dj8cg_e9dee2ef-6437-4963-8e27-da5fb9c75928/konnectivity-agent/0.log" limit=1024 Nov 03 14:10:54.879689 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.879658 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-pkhg9"] Nov 03 14:10:54.882141 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.882116 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-g28dw"] Nov 03 14:10:54.884916 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.884890 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wcb9f"] Nov 03 14:10:54.885781 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:54.885592 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47c859b2_a25b_45e5_acdc_0632d472c097.slice/crio-f02f60e185fa11f790a793a2b8a042d7b521b3c4a117b471dfeadb8a1f1cfc54 WatchSource:0}: Error finding container f02f60e185fa11f790a793a2b8a042d7b521b3c4a117b471dfeadb8a1f1cfc54: Status 404 returned error can't find the container with id f02f60e185fa11f790a793a2b8a042d7b521b3c4a117b471dfeadb8a1f1cfc54 Nov 03 14:10:54.886036 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:54.886015 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a412a1_3764_404b_9e6d_3a2bbb200781.slice/crio-13785db1c6914236913af97f8c8f3777a7a9ebf03e55a39f25ff3e08eb9c0328 WatchSource:0}: Error finding container 13785db1c6914236913af97f8c8f3777a7a9ebf03e55a39f25ff3e08eb9c0328: Status 404 returned error can't find the container with id 13785db1c6914236913af97f8c8f3777a7a9ebf03e55a39f25ff3e08eb9c0328 Nov 03 14:10:54.889067 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:54.889005 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d66aab2_4671_4bbd_a4cc_b9dd9c5a766d.slice/crio-7640e0f73cde1a06114a434eeb5ceb533b361171bd334ae3fff87ccbb55c03ff WatchSource:0}: Error finding container 7640e0f73cde1a06114a434eeb5ceb533b361171bd334ae3fff87ccbb55c03ff: Status 404 returned error can't find the container with id 7640e0f73cde1a06114a434eeb5ceb533b361171bd334ae3fff87ccbb55c03ff Nov 03 14:10:54.959764 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.959682 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-registry-tls\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:10:54.959764 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.959733 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36065516-87ab-4057-b198-439305606d12-service-ca-bundle\") pod \"router-default-6c7f54c9b7-t6wbf\" (UID: \"36065516-87ab-4057-b198-439305606d12\") " pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:10:54.960056 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.959789 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bdf183c4-a9ee-4ea8-ace1-50f307114328-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-v6pjk\" (UID: \"bdf183c4-a9ee-4ea8-ace1-50f307114328\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v6pjk" Nov 03 14:10:54.960056 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.959846 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36065516-87ab-4057-b198-439305606d12-metrics-certs\") pod \"router-default-6c7f54c9b7-t6wbf\" (UID: \"36065516-87ab-4057-b198-439305606d12\") " pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:10:54.960056 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.959917 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6f219713-6fd8-4580-b1c0-24e2a929c258-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zgd4k\" (UID: \"6f219713-6fd8-4580-b1c0-24e2a929c258\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zgd4k" Nov 03 14:10:54.960056 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.959978 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ea7bdafa-8d46-4b28-b93b-b73352b2e269-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-q79vl\" (UID: \"ea7bdafa-8d46-4b28-b93b-b73352b2e269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q79vl" Nov 03 14:10:54.960056 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:54.960009 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 14:10:54.960056 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:54.960045 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57647df665-9q52v: secret "image-registry-tls" not found Nov 03 14:10:54.960353 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:54.960106 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 14:10:54.960353 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:54.960120 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-registry-tls podName:fbfeab8a-b74e-4188-b565-b56442a01b79 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:55.960084805 +0000 UTC m=+31.576211385 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-registry-tls") pod "image-registry-57647df665-9q52v" (UID: "fbfeab8a-b74e-4188-b565-b56442a01b79") : secret "image-registry-tls" not found Nov 03 14:10:54.960353 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:54.960154 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea7bdafa-8d46-4b28-b93b-b73352b2e269-samples-operator-tls podName:ea7bdafa-8d46-4b28-b93b-b73352b2e269 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:55.960139596 +0000 UTC m=+31.576266200 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ea7bdafa-8d46-4b28-b93b-b73352b2e269-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-q79vl" (UID: "ea7bdafa-8d46-4b28-b93b-b73352b2e269") : secret "samples-operator-tls" not found Nov 03 14:10:54.960353 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:54.960219 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/36065516-87ab-4057-b198-439305606d12-service-ca-bundle podName:36065516-87ab-4057-b198-439305606d12 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:55.960209486 +0000 UTC m=+31.576336086 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/36065516-87ab-4057-b198-439305606d12-service-ca-bundle") pod "router-default-6c7f54c9b7-t6wbf" (UID: "36065516-87ab-4057-b198-439305606d12") : configmap references non-existent config key: service-ca.crt Nov 03 14:10:54.960353 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:54.960280 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 14:10:54.960353 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:54.960314 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdf183c4-a9ee-4ea8-ace1-50f307114328-cluster-monitoring-operator-tls podName:bdf183c4-a9ee-4ea8-ace1-50f307114328 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:55.960302146 +0000 UTC m=+31.576428726 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bdf183c4-a9ee-4ea8-ace1-50f307114328-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-v6pjk" (UID: "bdf183c4-a9ee-4ea8-ace1-50f307114328") : secret "cluster-monitoring-operator-tls" not found Nov 03 14:10:54.960602 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:54.960360 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 14:10:54.960602 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:54.960387 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36065516-87ab-4057-b198-439305606d12-metrics-certs podName:36065516-87ab-4057-b198-439305606d12 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:55.96037884 +0000 UTC m=+31.576505468 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/36065516-87ab-4057-b198-439305606d12-metrics-certs") pod "router-default-6c7f54c9b7-t6wbf" (UID: "36065516-87ab-4057-b198-439305606d12") : secret "router-metrics-certs-default" not found Nov 03 14:10:54.960602 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:54.960432 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 14:10:54.960602 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:54.960452 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f219713-6fd8-4580-b1c0-24e2a929c258-networking-console-plugin-cert podName:6f219713-6fd8-4580-b1c0-24e2a929c258 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:55.960446262 +0000 UTC m=+31.576572841 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6f219713-6fd8-4580-b1c0-24e2a929c258-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zgd4k" (UID: "6f219713-6fd8-4580-b1c0-24e2a929c258") : secret "networking-console-plugin-cert" not found Nov 03 14:10:54.998159 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:54.998129 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-v9mxv"] Nov 03 14:10:55.002911 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:55.002812 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-nvj5v"] Nov 03 14:10:55.017697 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:55.017078 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-5xzwg"] Nov 03 14:10:55.023827 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:55.023757 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-j66qt"] Nov 03 14:10:55.034884 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:55.034858 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd3829b5_04dc_4112_b2d9_087d136ba9ec.slice/crio-9c4f3c61cca7f03a33d5ad5e9a4a82825d334b9c05f2baebe6ab49f2dee6d18b WatchSource:0}: Error finding container 9c4f3c61cca7f03a33d5ad5e9a4a82825d334b9c05f2baebe6ab49f2dee6d18b: Status 404 returned error can't find the container with id 9c4f3c61cca7f03a33d5ad5e9a4a82825d334b9c05f2baebe6ab49f2dee6d18b Nov 03 14:10:55.055378 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:55.055129 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-rjz49"] Nov 03 14:10:55.061355 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:55.061327 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-registry-tls\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:10:55.061443 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:55.061415 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61060804-c519-490a-ac90-674792d50174-metrics-tls\") pod \"dns-default-2bk97\" (UID: \"61060804-c519-490a-ac90-674792d50174\") " pod="openshift-dns/dns-default-2bk97" Nov 03 14:10:55.061596 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:55.061581 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 14:10:55.061651 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:55.061641 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61060804-c519-490a-ac90-674792d50174-metrics-tls podName:61060804-c519-490a-ac90-674792d50174 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:56.06162408 +0000 UTC m=+31.677750679 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61060804-c519-490a-ac90-674792d50174-metrics-tls") pod "dns-default-2bk97" (UID: "61060804-c519-490a-ac90-674792d50174") : secret "dns-default-metrics-tls" not found Nov 03 14:10:55.061689 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:55.061641 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 14:10:55.061689 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:55.061662 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d7c596f48-qrcsg: secret "image-registry-tls" not found Nov 03 14:10:55.061968 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:55.061693 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-registry-tls podName:b3260ef2-8ff4-4b70-a3a0-e5386a27d210 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:56.061682828 +0000 UTC m=+31.677809422 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-registry-tls") pod "image-registry-6d7c596f48-qrcsg" (UID: "b3260ef2-8ff4-4b70-a3a0-e5386a27d210") : secret "image-registry-tls" not found Nov 03 14:10:55.061968 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:55.061772 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-slvhl"] Nov 03 14:10:55.064590 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:55.064435 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf085d57_4ae4_4add_9402_9ee1ce5d795a.slice/crio-e16a0074955a8d79bbe287de735b294a1d9747d2f59c22ca54ceebb7174f5a4a WatchSource:0}: Error finding container e16a0074955a8d79bbe287de735b294a1d9747d2f59c22ca54ceebb7174f5a4a: Status 404 returned error can't find the container with id e16a0074955a8d79bbe287de735b294a1d9747d2f59c22ca54ceebb7174f5a4a Nov 03 14:10:55.106032 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:55.106002 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qnjdm"] Nov 03 14:10:55.109262 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:55.109240 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63bfb9d0_9e87_44c6_a6f7_2f7b992e20f6.slice/crio-b6867239ea6208777242d582c882a81cd26834e80a64ddff473a7fdc1e89e0fd WatchSource:0}: Error finding container b6867239ea6208777242d582c882a81cd26834e80a64ddff473a7fdc1e89e0fd: Status 404 returned error can't find the container with id b6867239ea6208777242d582c882a81cd26834e80a64ddff473a7fdc1e89e0fd Nov 03 14:10:55.126960 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:55.126934 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rjz49" event={"ID":"840ddc67-db99-4d3f-ba2d-51632ac3912c","Type":"ContainerStarted","Data":"dcb4a147277102c0ee92ed924dc29e6304010e9c9c62749f58c0cc45c6f40640"} Nov 03 14:10:55.127922 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:55.127896 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-slvhl" event={"ID":"bf085d57-4ae4-4add-9402-9ee1ce5d795a","Type":"ContainerStarted","Data":"e16a0074955a8d79bbe287de735b294a1d9747d2f59c22ca54ceebb7174f5a4a"} Nov 03 14:10:55.129112 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:55.129094 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-wcb9f" event={"ID":"4d66aab2-4671-4bbd-a4cc-b9dd9c5a766d","Type":"ContainerStarted","Data":"b21f3f98019cf7564e0c77361c7506daee91c28e9a0d74a1bb23d887b6dcdf49"} Nov 03 14:10:55.129181 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:55.129123 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-wcb9f" event={"ID":"4d66aab2-4671-4bbd-a4cc-b9dd9c5a766d","Type":"ContainerStarted","Data":"7640e0f73cde1a06114a434eeb5ceb533b361171bd334ae3fff87ccbb55c03ff"} Nov 03 14:10:55.130513 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:55.130495 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-pkhg9" event={"ID":"47c859b2-a25b-45e5-acdc-0632d472c097","Type":"ContainerStarted","Data":"275a10365f479cc07ad7c55a14aa8578b4d6fdd4a105ac51a1a93ccf9d809bb7"} Nov 03 14:10:55.130610 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:55.130528 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-pkhg9" event={"ID":"47c859b2-a25b-45e5-acdc-0632d472c097","Type":"ContainerStarted","Data":"f02f60e185fa11f790a793a2b8a042d7b521b3c4a117b471dfeadb8a1f1cfc54"} Nov 03 14:10:55.131648 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:55.131631 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qnjdm" event={"ID":"63bfb9d0-9e87-44c6-a6f7-2f7b992e20f6","Type":"ContainerStarted","Data":"b6867239ea6208777242d582c882a81cd26834e80a64ddff473a7fdc1e89e0fd"} Nov 03 14:10:55.132653 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:55.132634 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nvj5v" event={"ID":"8d74c5e4-f631-4eae-84d5-84749b53ccf8","Type":"ContainerStarted","Data":"1da2eca1baf31392b9832da78c58b8a61a6e04b0b418d6920124b73c4c7d4b03"} Nov 03 14:10:55.133960 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:55.133939 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-g28dw" event={"ID":"00a412a1-3764-404b-9e6d-3a2bbb200781","Type":"ContainerStarted","Data":"1605cad58a5151032e6a072d41074b63f137e82716043c28e60dbdd2091c3efa"} Nov 03 14:10:55.134045 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:55.133977 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-g28dw" event={"ID":"00a412a1-3764-404b-9e6d-3a2bbb200781","Type":"ContainerStarted","Data":"13785db1c6914236913af97f8c8f3777a7a9ebf03e55a39f25ff3e08eb9c0328"} Nov 03 14:10:55.134954 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:55.134937 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-j66qt" event={"ID":"dd3829b5-04dc-4112-b2d9-087d136ba9ec","Type":"ContainerStarted","Data":"9c4f3c61cca7f03a33d5ad5e9a4a82825d334b9c05f2baebe6ab49f2dee6d18b"} Nov 03 14:10:55.135917 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:55.135902 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-v9mxv" event={"ID":"11d16990-ce0c-458f-ab1e-e9a3fed3221b","Type":"ContainerStarted","Data":"ccae5bf3c9ab1e801ea85c8e8dea88e7011d7b98b5015e975f65b768ea2eb6c3"} Nov 03 14:10:55.136926 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:55.136908 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" event={"ID":"307e48f6-4a66-4c9c-a4ff-848cce9d3e0e","Type":"ContainerStarted","Data":"db44b327584d75050b083495c3de9df2448c9f260f6100e73077383104187e6b"} Nov 03 14:10:55.179168 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:55.179123 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-wcb9f" podStartSLOduration=348.179108358 podStartE2EDuration="5m48.179108358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:05:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:10:55.178160564 +0000 UTC m=+30.794287164" watchObservedRunningTime="2026-11-03 14:10:55.179108358 +0000 UTC m=+30.795234962" Nov 03 14:10:55.220483 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:55.220406 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-g28dw" podStartSLOduration=348.22039052 podStartE2EDuration="5m48.22039052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:05:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:10:55.219956463 +0000 UTC m=+30.836083061" watchObservedRunningTime="2026-11-03 14:10:55.22039052 +0000 UTC m=+30.836517125" Nov 03 14:10:55.263501 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:55.263457 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-pkhg9" podStartSLOduration=348.263445862 podStartE2EDuration="5m48.263445862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:05:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:10:55.262680632 +0000 UTC m=+30.878807235" watchObservedRunningTime="2026-11-03 14:10:55.263445862 +0000 UTC m=+30.879572460" Nov 03 14:10:55.887949 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:55.887913 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkhr9" Nov 03 14:10:55.888412 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:55.888396 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xrr4" Nov 03 14:10:55.888593 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:55.888573 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc5nw" Nov 03 14:10:55.906446 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:55.906213 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 14:10:55.906446 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:55.906285 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-bgwlk\"" Nov 03 14:10:55.906446 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:55.906311 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ml487\"" Nov 03 14:10:55.906446 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:55.906325 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rmj5t\"" Nov 03 14:10:55.906446 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:55.906379 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 14:10:55.972893 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:55.971609 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-registry-tls\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:10:55.972893 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:55.971657 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36065516-87ab-4057-b198-439305606d12-service-ca-bundle\") pod \"router-default-6c7f54c9b7-t6wbf\" (UID: \"36065516-87ab-4057-b198-439305606d12\") " pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:10:55.972893 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:55.971710 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bdf183c4-a9ee-4ea8-ace1-50f307114328-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-v6pjk\" (UID: \"bdf183c4-a9ee-4ea8-ace1-50f307114328\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v6pjk" Nov 03 14:10:55.972893 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:55.971742 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36065516-87ab-4057-b198-439305606d12-metrics-certs\") pod \"router-default-6c7f54c9b7-t6wbf\" (UID: \"36065516-87ab-4057-b198-439305606d12\") " pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:10:55.972893 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:55.971815 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6f219713-6fd8-4580-b1c0-24e2a929c258-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zgd4k\" (UID: \"6f219713-6fd8-4580-b1c0-24e2a929c258\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zgd4k" Nov 03 14:10:55.972893 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:55.971902 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ea7bdafa-8d46-4b28-b93b-b73352b2e269-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-q79vl\" (UID: \"ea7bdafa-8d46-4b28-b93b-b73352b2e269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q79vl" Nov 03 14:10:55.972893 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:55.972033 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 14:10:55.972893 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:55.972090 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea7bdafa-8d46-4b28-b93b-b73352b2e269-samples-operator-tls podName:ea7bdafa-8d46-4b28-b93b-b73352b2e269 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:57.97207345 +0000 UTC m=+33.588200042 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ea7bdafa-8d46-4b28-b93b-b73352b2e269-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-q79vl" (UID: "ea7bdafa-8d46-4b28-b93b-b73352b2e269") : secret "samples-operator-tls" not found Nov 03 14:10:55.972893 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:55.972473 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 14:10:55.972893 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:55.972493 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57647df665-9q52v: secret "image-registry-tls" not found Nov 03 14:10:55.972893 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:55.972539 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-registry-tls podName:fbfeab8a-b74e-4188-b565-b56442a01b79 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:57.972524311 +0000 UTC m=+33.588650890 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-registry-tls") pod "image-registry-57647df665-9q52v" (UID: "fbfeab8a-b74e-4188-b565-b56442a01b79") : secret "image-registry-tls" not found Nov 03 14:10:55.972893 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:55.972614 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/36065516-87ab-4057-b198-439305606d12-service-ca-bundle podName:36065516-87ab-4057-b198-439305606d12 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:57.972603144 +0000 UTC m=+33.588729738 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/36065516-87ab-4057-b198-439305606d12-service-ca-bundle") pod "router-default-6c7f54c9b7-t6wbf" (UID: "36065516-87ab-4057-b198-439305606d12") : configmap references non-existent config key: service-ca.crt Nov 03 14:10:55.972893 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:55.972678 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 14:10:55.972893 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:55.972708 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdf183c4-a9ee-4ea8-ace1-50f307114328-cluster-monitoring-operator-tls podName:bdf183c4-a9ee-4ea8-ace1-50f307114328 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:57.972699746 +0000 UTC m=+33.588826341 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bdf183c4-a9ee-4ea8-ace1-50f307114328-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-v6pjk" (UID: "bdf183c4-a9ee-4ea8-ace1-50f307114328") : secret "cluster-monitoring-operator-tls" not found Nov 03 14:10:55.972893 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:55.972749 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 14:10:55.973782 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:55.972783 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36065516-87ab-4057-b198-439305606d12-metrics-certs podName:36065516-87ab-4057-b198-439305606d12 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:57.972777021 +0000 UTC m=+33.588903613 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/36065516-87ab-4057-b198-439305606d12-metrics-certs") pod "router-default-6c7f54c9b7-t6wbf" (UID: "36065516-87ab-4057-b198-439305606d12") : secret "router-metrics-certs-default" not found Nov 03 14:10:55.973782 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:55.972823 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 14:10:55.973782 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:55.972866 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f219713-6fd8-4580-b1c0-24e2a929c258-networking-console-plugin-cert podName:6f219713-6fd8-4580-b1c0-24e2a929c258 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:57.972857055 +0000 UTC m=+33.588983648 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6f219713-6fd8-4580-b1c0-24e2a929c258-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zgd4k" (UID: "6f219713-6fd8-4580-b1c0-24e2a929c258") : secret "networking-console-plugin-cert" not found Nov 03 14:10:56.074056 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:56.073095 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-registry-tls\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:10:56.074056 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:56.073216 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61060804-c519-490a-ac90-674792d50174-metrics-tls\") pod \"dns-default-2bk97\" (UID: \"61060804-c519-490a-ac90-674792d50174\") " pod="openshift-dns/dns-default-2bk97" Nov 03 14:10:56.074056 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:56.073932 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 14:10:56.074056 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:56.073989 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61060804-c519-490a-ac90-674792d50174-metrics-tls podName:61060804-c519-490a-ac90-674792d50174 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:58.073974895 +0000 UTC m=+33.690101488 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61060804-c519-490a-ac90-674792d50174-metrics-tls") pod "dns-default-2bk97" (UID: "61060804-c519-490a-ac90-674792d50174") : secret "dns-default-metrics-tls" not found Nov 03 14:10:56.074568 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:56.074427 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 14:10:56.074568 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:56.074462 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d7c596f48-qrcsg: secret "image-registry-tls" not found Nov 03 14:10:56.074568 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:56.074522 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-registry-tls podName:b3260ef2-8ff4-4b70-a3a0-e5386a27d210 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:58.074509594 +0000 UTC m=+33.690636186 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-registry-tls") pod "image-registry-6d7c596f48-qrcsg" (UID: "b3260ef2-8ff4-4b70-a3a0-e5386a27d210") : secret "image-registry-tls" not found Nov 03 14:10:57.993869 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:57.993506 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6f219713-6fd8-4580-b1c0-24e2a929c258-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zgd4k\" (UID: \"6f219713-6fd8-4580-b1c0-24e2a929c258\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zgd4k" Nov 03 14:10:57.993869 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:57.993598 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ea7bdafa-8d46-4b28-b93b-b73352b2e269-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-q79vl\" (UID: \"ea7bdafa-8d46-4b28-b93b-b73352b2e269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q79vl" Nov 03 14:10:57.993869 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:57.993658 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 14:10:57.993869 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:57.993704 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-registry-tls\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:10:57.993869 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:57.993720 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f219713-6fd8-4580-b1c0-24e2a929c258-networking-console-plugin-cert podName:6f219713-6fd8-4580-b1c0-24e2a929c258 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:01.993705503 +0000 UTC m=+37.609832082 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6f219713-6fd8-4580-b1c0-24e2a929c258-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zgd4k" (UID: "6f219713-6fd8-4580-b1c0-24e2a929c258") : secret "networking-console-plugin-cert" not found Nov 03 14:10:57.993869 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:57.993735 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 14:10:57.993869 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:57.993759 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36065516-87ab-4057-b198-439305606d12-service-ca-bundle\") pod \"router-default-6c7f54c9b7-t6wbf\" (UID: \"36065516-87ab-4057-b198-439305606d12\") " pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:10:57.993869 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:57.993780 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 14:10:57.993869 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:57.993791 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea7bdafa-8d46-4b28-b93b-b73352b2e269-samples-operator-tls podName:ea7bdafa-8d46-4b28-b93b-b73352b2e269 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:01.993774963 +0000 UTC m=+37.609901558 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ea7bdafa-8d46-4b28-b93b-b73352b2e269-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-q79vl" (UID: "ea7bdafa-8d46-4b28-b93b-b73352b2e269") : secret "samples-operator-tls" not found Nov 03 14:10:57.993869 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:57.993794 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57647df665-9q52v: secret "image-registry-tls" not found Nov 03 14:10:57.993869 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:57.993814 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/36065516-87ab-4057-b198-439305606d12-service-ca-bundle podName:36065516-87ab-4057-b198-439305606d12 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:01.99380674 +0000 UTC m=+37.609933374 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/36065516-87ab-4057-b198-439305606d12-service-ca-bundle") pod "router-default-6c7f54c9b7-t6wbf" (UID: "36065516-87ab-4057-b198-439305606d12") : configmap references non-existent config key: service-ca.crt Nov 03 14:10:57.993869 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:57.993861 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-registry-tls podName:fbfeab8a-b74e-4188-b565-b56442a01b79 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:01.993849075 +0000 UTC m=+37.609975655 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-registry-tls") pod "image-registry-57647df665-9q52v" (UID: "fbfeab8a-b74e-4188-b565-b56442a01b79") : secret "image-registry-tls" not found Nov 03 14:10:57.993869 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:57.993857 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bdf183c4-a9ee-4ea8-ace1-50f307114328-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-v6pjk\" (UID: \"bdf183c4-a9ee-4ea8-ace1-50f307114328\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v6pjk" Nov 03 14:10:57.995652 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:57.993907 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 14:10:57.995652 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:57.994012 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdf183c4-a9ee-4ea8-ace1-50f307114328-cluster-monitoring-operator-tls podName:bdf183c4-a9ee-4ea8-ace1-50f307114328 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:01.993999825 +0000 UTC m=+37.610126409 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bdf183c4-a9ee-4ea8-ace1-50f307114328-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-v6pjk" (UID: "bdf183c4-a9ee-4ea8-ace1-50f307114328") : secret "cluster-monitoring-operator-tls" not found Nov 03 14:10:57.995652 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:57.994049 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36065516-87ab-4057-b198-439305606d12-metrics-certs\") pod \"router-default-6c7f54c9b7-t6wbf\" (UID: \"36065516-87ab-4057-b198-439305606d12\") " pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:10:57.995652 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:57.994205 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 14:10:57.995652 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:57.994250 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36065516-87ab-4057-b198-439305606d12-metrics-certs podName:36065516-87ab-4057-b198-439305606d12 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:01.994238267 +0000 UTC m=+37.610364848 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/36065516-87ab-4057-b198-439305606d12-metrics-certs") pod "router-default-6c7f54c9b7-t6wbf" (UID: "36065516-87ab-4057-b198-439305606d12") : secret "router-metrics-certs-default" not found Nov 03 14:10:58.094572 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:58.094542 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-registry-tls\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:10:58.094814 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:58.094623 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61060804-c519-490a-ac90-674792d50174-metrics-tls\") pod \"dns-default-2bk97\" (UID: \"61060804-c519-490a-ac90-674792d50174\") " pod="openshift-dns/dns-default-2bk97" Nov 03 14:10:58.094814 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:58.094769 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 14:10:58.094814 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:58.094778 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 14:10:58.094814 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:58.094797 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d7c596f48-qrcsg: secret "image-registry-tls" not found Nov 03 14:10:58.095124 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:58.094823 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61060804-c519-490a-ac90-674792d50174-metrics-tls podName:61060804-c519-490a-ac90-674792d50174 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:02.09480599 +0000 UTC m=+37.710932586 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61060804-c519-490a-ac90-674792d50174-metrics-tls") pod "dns-default-2bk97" (UID: "61060804-c519-490a-ac90-674792d50174") : secret "dns-default-metrics-tls" not found Nov 03 14:10:58.095124 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:58.094867 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-registry-tls podName:b3260ef2-8ff4-4b70-a3a0-e5386a27d210 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:02.094848368 +0000 UTC m=+37.710974961 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-registry-tls") pod "image-registry-6d7c596f48-qrcsg" (UID: "b3260ef2-8ff4-4b70-a3a0-e5386a27d210") : secret "image-registry-tls" not found Nov 03 14:10:58.154241 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:58.154200 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-v9mxv" event={"ID":"11d16990-ce0c-458f-ab1e-e9a3fed3221b","Type":"ContainerStarted","Data":"795ff33413e0a3514537c6d0a7248734f7e84210504d991884f41e39d1b24e01"} Nov 03 14:10:58.198523 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:58.197787 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-v9mxv" podStartSLOduration=320.224149762 podStartE2EDuration="5m22.197768282s" totalImagesPullingTime="1.97361852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:05:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:10:58.196989991 +0000 UTC m=+33.813116593" watchObservedRunningTime="2026-11-03 14:10:58.197768282 +0000 UTC m=+33.813894885" Nov 03 14:10:58.700981 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:58.700905 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f52cae01-7951-4d9e-9a57-9ba5bcdf6857-metrics-certs\") pod \"network-metrics-daemon-tc5nw\" (UID: \"f52cae01-7951-4d9e-9a57-9ba5bcdf6857\") " pod="openshift-multus/network-metrics-daemon-tc5nw" Nov 03 14:10:58.701148 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:58.701004 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef43449c-b7dc-4e84-b785-63201f80d4da-original-pull-secret\") pod \"global-pull-secret-syncer-2xrr4\" (UID: \"ef43449c-b7dc-4e84-b785-63201f80d4da\") " pod="kube-system/global-pull-secret-syncer-2xrr4" Nov 03 14:10:58.701623 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:58.701597 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 14:10:58.701746 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:10:58.701671 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f52cae01-7951-4d9e-9a57-9ba5bcdf6857-metrics-certs podName:f52cae01-7951-4d9e-9a57-9ba5bcdf6857 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:30.701651158 +0000 UTC m=+66.317777737 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f52cae01-7951-4d9e-9a57-9ba5bcdf6857-metrics-certs") pod "network-metrics-daemon-tc5nw" (UID: "f52cae01-7951-4d9e-9a57-9ba5bcdf6857") : secret "metrics-daemon-secret" not found Nov 03 14:10:58.705397 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:58.705366 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef43449c-b7dc-4e84-b785-63201f80d4da-original-pull-secret\") pod \"global-pull-secret-syncer-2xrr4\" (UID: \"ef43449c-b7dc-4e84-b785-63201f80d4da\") " pod="kube-system/global-pull-secret-syncer-2xrr4" Nov 03 14:10:58.802471 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:58.801759 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkjrd\" (UniqueName: \"kubernetes.io/projected/328dde79-0c85-47a3-b3f8-38d8cc8614c0-kube-api-access-dkjrd\") pod \"network-check-target-xkhr9\" (UID: \"328dde79-0c85-47a3-b3f8-38d8cc8614c0\") " pod="openshift-network-diagnostics/network-check-target-xkhr9" Nov 03 14:10:58.805675 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:58.805626 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkjrd\" (UniqueName: \"kubernetes.io/projected/328dde79-0c85-47a3-b3f8-38d8cc8614c0-kube-api-access-dkjrd\") pod \"network-check-target-xkhr9\" (UID: \"328dde79-0c85-47a3-b3f8-38d8cc8614c0\") " pod="openshift-network-diagnostics/network-check-target-xkhr9" Nov 03 14:10:58.925032 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:58.925002 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xkhr9" Nov 03 14:10:58.945529 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:58.945353 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xrr4" Nov 03 14:10:59.160540 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:59.160500 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/0.log" Nov 03 14:10:59.161127 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:59.160549 2577 generic.go:356] "Generic (PLEG): container finished" podID="307e48f6-4a66-4c9c-a4ff-848cce9d3e0e" containerID="c6f55c3224955cc474412033fcce30efaf605f6b72941b34214a1b65b5fcfbee" exitCode=255 Nov 03 14:10:59.161958 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:59.161327 2577 scope.go:122] "RemoveContainer" containerID="c6f55c3224955cc474412033fcce30efaf605f6b72941b34214a1b65b5fcfbee" Nov 03 14:10:59.161958 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:59.161599 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" event={"ID":"307e48f6-4a66-4c9c-a4ff-848cce9d3e0e","Type":"ContainerDied","Data":"c6f55c3224955cc474412033fcce30efaf605f6b72941b34214a1b65b5fcfbee"} Nov 03 14:10:59.260772 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:59.260749 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xkhr9"] Nov 03 14:10:59.293979 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:10:59.293950 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2xrr4"] Nov 03 14:10:59.321173 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:10:59.321147 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod328dde79_0c85_47a3_b3f8_38d8cc8614c0.slice/crio-9876113882c1611251a2fc38d5a386ff1f49e928e66cd1a51b21a3d5a08542ae WatchSource:0}: Error finding container 9876113882c1611251a2fc38d5a386ff1f49e928e66cd1a51b21a3d5a08542ae: Status 404 returned error can't find the container with id 9876113882c1611251a2fc38d5a386ff1f49e928e66cd1a51b21a3d5a08542ae Nov 03 14:11:00.164704 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.164624 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-j66qt" event={"ID":"dd3829b5-04dc-4112-b2d9-087d136ba9ec","Type":"ContainerStarted","Data":"586cf47171ee95b4f449f8b14c62e58862cde12b2db560a5917d6a54dff60ed1"} Nov 03 14:11:00.165884 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.165859 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2xrr4" event={"ID":"ef43449c-b7dc-4e84-b785-63201f80d4da","Type":"ContainerStarted","Data":"ae33d6534d669fa34314d880c5da457d96e430d11244450ad159fd464d82ea95"} Nov 03 14:11:00.168685 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.168667 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/1.log" Nov 03 14:11:00.169095 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.169080 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/0.log" Nov 03 14:11:00.169162 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.169108 2577 generic.go:356] "Generic (PLEG): container finished" podID="307e48f6-4a66-4c9c-a4ff-848cce9d3e0e" containerID="d87a44a09d0ad13a61108d4537be53ab5164f5519ade849da1c1ba1e459d0490" exitCode=255 Nov 03 14:11:00.169219 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.169159 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" event={"ID":"307e48f6-4a66-4c9c-a4ff-848cce9d3e0e","Type":"ContainerDied","Data":"d87a44a09d0ad13a61108d4537be53ab5164f5519ade849da1c1ba1e459d0490"} Nov 03 14:11:00.169219 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.169192 2577 scope.go:122] "RemoveContainer" containerID="c6f55c3224955cc474412033fcce30efaf605f6b72941b34214a1b65b5fcfbee" Nov 03 14:11:00.169393 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.169369 2577 scope.go:122] "RemoveContainer" containerID="d87a44a09d0ad13a61108d4537be53ab5164f5519ade849da1c1ba1e459d0490" Nov 03 14:11:00.169599 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:11:00.169574 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5xzwg_openshift-console-operator(307e48f6-4a66-4c9c-a4ff-848cce9d3e0e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" podUID="307e48f6-4a66-4c9c-a4ff-848cce9d3e0e" Nov 03 14:11:00.170392 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.170373 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xkhr9" event={"ID":"328dde79-0c85-47a3-b3f8-38d8cc8614c0","Type":"ContainerStarted","Data":"9876113882c1611251a2fc38d5a386ff1f49e928e66cd1a51b21a3d5a08542ae"} Nov 03 14:11:00.171739 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.171718 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-slvhl" event={"ID":"bf085d57-4ae4-4add-9402-9ee1ce5d795a","Type":"ContainerStarted","Data":"0f629cd99feb4211fca577a43c8e18f37eda996a790d401fb93cddb3f3122c2b"} Nov 03 14:11:00.209321 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.209272 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-j66qt" podStartSLOduration=319.420102263 podStartE2EDuration="5m24.209256717s" totalImagesPullingTime="4.789154454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:05:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:00.208682754 +0000 UTC m=+35.824809355" watchObservedRunningTime="2026-11-03 14:11:00.209256717 +0000 UTC m=+35.825383320" Nov 03 14:11:00.297182 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.297133 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-slvhl" podStartSLOduration=319.527350299 podStartE2EDuration="5m24.297114091s" totalImagesPullingTime="4.769763792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:05:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:00.296447493 +0000 UTC m=+35.912574095" watchObservedRunningTime="2026-11-03 14:11:00.297114091 +0000 UTC m=+35.913240692" Nov 03 14:11:00.596434 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.596398 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-sm2jg"] Nov 03 14:11:00.615634 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.615604 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:00.625300 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.625278 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sm2jg"] Nov 03 14:11:00.634047 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.634025 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 14:11:00.636211 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.636189 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 14:11:00.636676 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.636652 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hlrhs\"" Nov 03 14:11:00.665593 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.665557 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-fzk44-mlg6z"] Nov 03 14:11:00.684768 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.684743 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fzk44-mlg6z" Nov 03 14:11:00.693558 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.693496 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-fzk44-mlg6z"] Nov 03 14:11:00.721765 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.721732 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-data-volume\") pod \"insights-runtime-extractor-sm2jg\" (UID: \"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee\") " pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:00.721926 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.721779 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sm2jg\" (UID: \"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee\") " pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:00.721926 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.721812 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-crio-socket\") pod \"insights-runtime-extractor-sm2jg\" (UID: \"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee\") " pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:00.721926 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.721902 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sm2jg\" (UID: \"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee\") " pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:00.722097 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.721938 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdggk\" (UniqueName: \"kubernetes.io/projected/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-kube-api-access-hdggk\") pod \"insights-runtime-extractor-sm2jg\" (UID: \"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee\") " pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:00.726613 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.726470 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-qgp8d\"" Nov 03 14:11:00.794921 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.793418 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-c8hlq"] Nov 03 14:11:00.817017 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.816963 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-pn8kf"] Nov 03 14:11:00.818271 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.817315 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-c8hlq" Nov 03 14:11:00.823376 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.823175 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-crio-socket\") pod \"insights-runtime-extractor-sm2jg\" (UID: \"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee\") " pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:00.823376 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.823214 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d-service-ca-bundle\") pod \"periodic-gathering-fzk44-mlg6z\" (UID: \"7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d\") " pod="openshift-insights/periodic-gathering-fzk44-mlg6z" Nov 03 14:11:00.823376 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.823288 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d-archives-path\") pod \"periodic-gathering-fzk44-mlg6z\" (UID: \"7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d\") " pod="openshift-insights/periodic-gathering-fzk44-mlg6z" Nov 03 14:11:00.823376 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.823325 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d-trusted-ca-bundle\") pod \"periodic-gathering-fzk44-mlg6z\" (UID: \"7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d\") " pod="openshift-insights/periodic-gathering-fzk44-mlg6z" Nov 03 14:11:00.824861 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.823705 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sm2jg\" (UID: \"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee\") " pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:00.824861 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.823782 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdggk\" (UniqueName: \"kubernetes.io/projected/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-kube-api-access-hdggk\") pod \"insights-runtime-extractor-sm2jg\" (UID: \"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee\") " pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:00.824861 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.823861 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-data-volume\") pod \"insights-runtime-extractor-sm2jg\" (UID: \"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee\") " pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:00.824861 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.823887 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5m26\" (UniqueName: \"kubernetes.io/projected/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d-kube-api-access-m5m26\") pod \"periodic-gathering-fzk44-mlg6z\" (UID: \"7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d\") " pod="openshift-insights/periodic-gathering-fzk44-mlg6z" Nov 03 14:11:00.824861 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.823934 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sm2jg\" (UID: \"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee\") " pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:00.824861 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:11:00.824032 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 14:11:00.824861 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:11:00.824084 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-kube-rbac-proxy-cm podName:3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee nodeName:}" failed. No retries permitted until 2026-11-03 14:11:01.324067486 +0000 UTC m=+36.940194080 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sm2jg" (UID: "3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee") : configmap "kube-rbac-proxy" not found Nov 03 14:11:00.824861 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.824140 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-crio-socket\") pod \"insights-runtime-extractor-sm2jg\" (UID: \"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee\") " pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:00.824861 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:11:00.824282 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 14:11:00.824861 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:11:00.824315 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-insights-runtime-extractor-tls podName:3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee nodeName:}" failed. No retries permitted until 2026-11-03 14:11:01.324304749 +0000 UTC m=+36.940431341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sm2jg" (UID: "3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee") : secret "insights-runtime-extractor-tls" not found Nov 03 14:11:00.824861 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.824817 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-data-volume\") pod \"insights-runtime-extractor-sm2jg\" (UID: \"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee\") " pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:00.837110 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.836848 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-h44zq\"" Nov 03 14:11:00.837110 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.836978 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 14:11:00.837110 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.837042 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 14:11:00.845600 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.845566 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-c8hlq"] Nov 03 14:11:00.845732 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.845610 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-pn8kf"] Nov 03 14:11:00.845779 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.845764 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pn8kf" Nov 03 14:11:00.857381 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.857311 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdggk\" (UniqueName: \"kubernetes.io/projected/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-kube-api-access-hdggk\") pod \"insights-runtime-extractor-sm2jg\" (UID: \"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee\") " pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:00.927546 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.926084 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d-archives-path\") pod \"periodic-gathering-fzk44-mlg6z\" (UID: \"7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d\") " pod="openshift-insights/periodic-gathering-fzk44-mlg6z" Nov 03 14:11:00.927546 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.926139 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d-trusted-ca-bundle\") pod \"periodic-gathering-fzk44-mlg6z\" (UID: \"7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d\") " pod="openshift-insights/periodic-gathering-fzk44-mlg6z" Nov 03 14:11:00.927546 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.926256 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5m26\" (UniqueName: \"kubernetes.io/projected/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d-kube-api-access-m5m26\") pod \"periodic-gathering-fzk44-mlg6z\" (UID: \"7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d\") " pod="openshift-insights/periodic-gathering-fzk44-mlg6z" Nov 03 14:11:00.927546 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.926306 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr24x\" (UniqueName: \"kubernetes.io/projected/1da80d21-c62b-44ec-a3fe-64633b5de401-kube-api-access-nr24x\") pod \"migrator-66df5d496b-c8hlq\" (UID: \"1da80d21-c62b-44ec-a3fe-64633b5de401\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-c8hlq" Nov 03 14:11:00.927546 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.926364 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d-service-ca-bundle\") pod \"periodic-gathering-fzk44-mlg6z\" (UID: \"7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d\") " pod="openshift-insights/periodic-gathering-fzk44-mlg6z" Nov 03 14:11:00.927546 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.926937 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d-service-ca-bundle\") pod \"periodic-gathering-fzk44-mlg6z\" (UID: \"7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d\") " pod="openshift-insights/periodic-gathering-fzk44-mlg6z" Nov 03 14:11:00.927546 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.927506 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d-archives-path\") pod \"periodic-gathering-fzk44-mlg6z\" (UID: \"7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d\") " pod="openshift-insights/periodic-gathering-fzk44-mlg6z" Nov 03 14:11:00.927822 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.927693 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d-trusted-ca-bundle\") pod \"periodic-gathering-fzk44-mlg6z\" (UID: \"7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d\") " pod="openshift-insights/periodic-gathering-fzk44-mlg6z" Nov 03 14:11:00.949445 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:00.949402 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5m26\" (UniqueName: \"kubernetes.io/projected/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d-kube-api-access-m5m26\") pod \"periodic-gathering-fzk44-mlg6z\" (UID: \"7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d\") " pod="openshift-insights/periodic-gathering-fzk44-mlg6z" Nov 03 14:11:01.016387 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:01.016360 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fzk44-mlg6z" Nov 03 14:11:01.027343 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:01.026876 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5qqj\" (UniqueName: \"kubernetes.io/projected/60d2d1a6-a89f-4a9a-acb9-e85ce2c8f6ce-kube-api-access-w5qqj\") pod \"migrator-66df5d496b-pn8kf\" (UID: \"60d2d1a6-a89f-4a9a-acb9-e85ce2c8f6ce\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pn8kf" Nov 03 14:11:01.027343 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:01.027035 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr24x\" (UniqueName: \"kubernetes.io/projected/1da80d21-c62b-44ec-a3fe-64633b5de401-kube-api-access-nr24x\") pod \"migrator-66df5d496b-c8hlq\" (UID: \"1da80d21-c62b-44ec-a3fe-64633b5de401\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-c8hlq" Nov 03 14:11:01.049474 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:01.049424 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr24x\" (UniqueName: \"kubernetes.io/projected/1da80d21-c62b-44ec-a3fe-64633b5de401-kube-api-access-nr24x\") pod \"migrator-66df5d496b-c8hlq\" (UID: \"1da80d21-c62b-44ec-a3fe-64633b5de401\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-c8hlq" Nov 03 14:11:01.128898 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:01.128804 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5qqj\" (UniqueName: \"kubernetes.io/projected/60d2d1a6-a89f-4a9a-acb9-e85ce2c8f6ce-kube-api-access-w5qqj\") pod \"migrator-66df5d496b-pn8kf\" (UID: \"60d2d1a6-a89f-4a9a-acb9-e85ce2c8f6ce\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pn8kf" Nov 03 14:11:01.154827 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:01.154643 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-c8hlq" Nov 03 14:11:01.156357 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:01.156302 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5qqj\" (UniqueName: \"kubernetes.io/projected/60d2d1a6-a89f-4a9a-acb9-e85ce2c8f6ce-kube-api-access-w5qqj\") pod \"migrator-66df5d496b-pn8kf\" (UID: \"60d2d1a6-a89f-4a9a-acb9-e85ce2c8f6ce\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pn8kf" Nov 03 14:11:01.177180 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:01.177142 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nvj5v" event={"ID":"8d74c5e4-f631-4eae-84d5-84749b53ccf8","Type":"ContainerStarted","Data":"8634cdc639b1add0ed6a8e211015302fa0ddc90759373b2fd74a373e9448f40e"} Nov 03 14:11:01.179122 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:01.179102 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/1.log" Nov 03 14:11:01.179495 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:01.179476 2577 scope.go:122] "RemoveContainer" containerID="d87a44a09d0ad13a61108d4537be53ab5164f5519ade849da1c1ba1e459d0490" Nov 03 14:11:01.179701 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:11:01.179679 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5xzwg_openshift-console-operator(307e48f6-4a66-4c9c-a4ff-848cce9d3e0e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" podUID="307e48f6-4a66-4c9c-a4ff-848cce9d3e0e" Nov 03 14:11:01.181376 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:01.181020 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xkhr9" event={"ID":"328dde79-0c85-47a3-b3f8-38d8cc8614c0","Type":"ContainerStarted","Data":"aa06d1214eb22dfb13cba08012dcd109b0bf12a169d5bceddea1c5945fbe3b4f"} Nov 03 14:11:01.181376 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:01.181079 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xkhr9" Nov 03 14:11:01.182968 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:01.182594 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rjz49" event={"ID":"840ddc67-db99-4d3f-ba2d-51632ac3912c","Type":"ContainerStarted","Data":"f7676d49927fc4e8efc1b6277c1b5c4db1d4f5b3f40c9036bcf23193612d3c12"} Nov 03 14:11:01.185111 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:01.184753 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qnjdm" event={"ID":"63bfb9d0-9e87-44c6-a6f7-2f7b992e20f6","Type":"ContainerStarted","Data":"43677b0173f5ce06aa63e802b2e0a273e6222ce1d16f037f6d5dbae31543fc22"} Nov 03 14:11:01.189402 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:01.189386 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pn8kf" Nov 03 14:11:01.223922 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:01.222881 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nvj5v" podStartSLOduration=319.629737478 podStartE2EDuration="5m25.222861313s" totalImagesPullingTime="5.593123835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:05:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:01.221719673 +0000 UTC m=+36.837846275" watchObservedRunningTime="2026-11-03 14:11:01.222861313 +0000 UTC m=+36.838987915" Nov 03 14:11:01.268595 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:01.267995 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xkhr9" podStartSLOduration=34.718723575 podStartE2EDuration="36.267977004s" totalImagesPullingTime="1.549253429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:10:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:01.266359233 +0000 UTC m=+36.882485836" watchObservedRunningTime="2026-11-03 14:11:01.267977004 +0000 UTC m=+36.884103606" Nov 03 14:11:01.331220 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:01.331189 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sm2jg\" (UID: \"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee\") " pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:01.331470 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:01.331452 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sm2jg\" (UID: \"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee\") " pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:01.331591 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:11:01.331578 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 14:11:01.331650 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:11:01.331638 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-kube-rbac-proxy-cm podName:3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee nodeName:}" failed. No retries permitted until 2026-11-03 14:11:02.331620495 +0000 UTC m=+37.947747091 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sm2jg" (UID: "3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee") : configmap "kube-rbac-proxy" not found Nov 03 14:11:01.334131 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:11:01.331981 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 14:11:01.334131 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:11:01.332046 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-insights-runtime-extractor-tls podName:3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee nodeName:}" failed. No retries permitted until 2026-11-03 14:11:02.332029582 +0000 UTC m=+37.948156166 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sm2jg" (UID: "3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee") : secret "insights-runtime-extractor-tls" not found Nov 03 14:11:01.345393 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:01.345318 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-fzk44-mlg6z"] Nov 03 14:11:01.349728 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:11:01.349686 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fe0ba2e_a1f0_41c4_b822_ffdc9c50ae4d.slice/crio-8d03b1024d60e73a8391f33068de1832e67c70d38283aa74f1366514ff8452c5 WatchSource:0}: Error finding container 8d03b1024d60e73a8391f33068de1832e67c70d38283aa74f1366514ff8452c5: Status 404 returned error can't find the container with id 8d03b1024d60e73a8391f33068de1832e67c70d38283aa74f1366514ff8452c5 Nov 03 14:11:01.364697 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:01.364643 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rjz49" podStartSLOduration=319.794726466 podStartE2EDuration="5m25.364626203s" totalImagesPullingTime="5.569899737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:05:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:01.362877158 +0000 UTC m=+36.979003759" watchObservedRunningTime="2026-11-03 14:11:01.364626203 +0000 UTC m=+36.980752805" Nov 03 14:11:01.407368 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:01.407325 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qnjdm" podStartSLOduration=1.9001405120000001 podStartE2EDuration="7.407312506s" totalImagesPullingTime="5.507171994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:10:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:01.406686239 +0000 UTC m=+37.022812840" watchObservedRunningTime="2026-11-03 14:11:01.407312506 +0000 UTC m=+37.023439106" Nov 03 14:11:01.486903 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:01.486797 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-c8hlq"] Nov 03 14:11:01.494736 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:11:01.494704 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1da80d21_c62b_44ec_a3fe_64633b5de401.slice/crio-430a1e4878dba2ce0ac77cbc20ec0e6f83bd85eacc21aa2aeb477f543173caca WatchSource:0}: Error finding container 430a1e4878dba2ce0ac77cbc20ec0e6f83bd85eacc21aa2aeb477f543173caca: Status 404 returned error can't find the container with id 430a1e4878dba2ce0ac77cbc20ec0e6f83bd85eacc21aa2aeb477f543173caca Nov 03 14:11:01.541271 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:01.541211 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-pn8kf"] Nov 03 14:11:02.038037 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:02.038002 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-registry-tls\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:11:02.038037 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:02.038046 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36065516-87ab-4057-b198-439305606d12-service-ca-bundle\") pod \"router-default-6c7f54c9b7-t6wbf\" (UID: \"36065516-87ab-4057-b198-439305606d12\") " pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:11:02.038290 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:02.038093 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bdf183c4-a9ee-4ea8-ace1-50f307114328-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-v6pjk\" (UID: \"bdf183c4-a9ee-4ea8-ace1-50f307114328\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v6pjk" Nov 03 14:11:02.038290 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:11:02.038180 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 14:11:02.038290 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:11:02.038187 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 14:11:02.038290 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:11:02.038210 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57647df665-9q52v: secret "image-registry-tls" not found Nov 03 14:11:02.038290 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:11:02.038249 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdf183c4-a9ee-4ea8-ace1-50f307114328-cluster-monitoring-operator-tls podName:bdf183c4-a9ee-4ea8-ace1-50f307114328 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:10.038230782 +0000 UTC m=+45.654357370 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bdf183c4-a9ee-4ea8-ace1-50f307114328-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-v6pjk" (UID: "bdf183c4-a9ee-4ea8-ace1-50f307114328") : secret "cluster-monitoring-operator-tls" not found Nov 03 14:11:02.038290 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:02.038275 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36065516-87ab-4057-b198-439305606d12-metrics-certs\") pod \"router-default-6c7f54c9b7-t6wbf\" (UID: \"36065516-87ab-4057-b198-439305606d12\") " pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:11:02.038608 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:02.038356 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6f219713-6fd8-4580-b1c0-24e2a929c258-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zgd4k\" (UID: \"6f219713-6fd8-4580-b1c0-24e2a929c258\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zgd4k" Nov 03 14:11:02.038608 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:02.038439 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ea7bdafa-8d46-4b28-b93b-b73352b2e269-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-q79vl\" (UID: \"ea7bdafa-8d46-4b28-b93b-b73352b2e269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q79vl" Nov 03 14:11:02.038608 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:11:02.038552 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 14:11:02.038608 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:11:02.038596 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea7bdafa-8d46-4b28-b93b-b73352b2e269-samples-operator-tls podName:ea7bdafa-8d46-4b28-b93b-b73352b2e269 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:10.038580387 +0000 UTC m=+45.654706982 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ea7bdafa-8d46-4b28-b93b-b73352b2e269-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-q79vl" (UID: "ea7bdafa-8d46-4b28-b93b-b73352b2e269") : secret "samples-operator-tls" not found Nov 03 14:11:02.038802 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:11:02.038616 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/36065516-87ab-4057-b198-439305606d12-service-ca-bundle podName:36065516-87ab-4057-b198-439305606d12 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:10.038602586 +0000 UTC m=+45.654729170 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/36065516-87ab-4057-b198-439305606d12-service-ca-bundle") pod "router-default-6c7f54c9b7-t6wbf" (UID: "36065516-87ab-4057-b198-439305606d12") : configmap references non-existent config key: service-ca.crt Nov 03 14:11:02.038802 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:11:02.038652 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-registry-tls podName:fbfeab8a-b74e-4188-b565-b56442a01b79 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:10.038634053 +0000 UTC m=+45.654760637 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-registry-tls") pod "image-registry-57647df665-9q52v" (UID: "fbfeab8a-b74e-4188-b565-b56442a01b79") : secret "image-registry-tls" not found Nov 03 14:11:02.038802 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:11:02.038655 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 14:11:02.038802 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:11:02.038692 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36065516-87ab-4057-b198-439305606d12-metrics-certs podName:36065516-87ab-4057-b198-439305606d12 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:10.038680926 +0000 UTC m=+45.654807507 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/36065516-87ab-4057-b198-439305606d12-metrics-certs") pod "router-default-6c7f54c9b7-t6wbf" (UID: "36065516-87ab-4057-b198-439305606d12") : secret "router-metrics-certs-default" not found Nov 03 14:11:02.038802 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:11:02.038711 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 14:11:02.038802 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:11:02.038747 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f219713-6fd8-4580-b1c0-24e2a929c258-networking-console-plugin-cert podName:6f219713-6fd8-4580-b1c0-24e2a929c258 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:10.038737143 +0000 UTC m=+45.654863724 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6f219713-6fd8-4580-b1c0-24e2a929c258-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zgd4k" (UID: "6f219713-6fd8-4580-b1c0-24e2a929c258") : secret "networking-console-plugin-cert" not found Nov 03 14:11:02.139291 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:02.139256 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-registry-tls\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:11:02.139628 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:02.139323 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61060804-c519-490a-ac90-674792d50174-metrics-tls\") pod \"dns-default-2bk97\" (UID: \"61060804-c519-490a-ac90-674792d50174\") " pod="openshift-dns/dns-default-2bk97" Nov 03 14:11:02.139628 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:11:02.139439 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 14:11:02.139628 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:11:02.139458 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 14:11:02.139628 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:11:02.139464 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d7c596f48-qrcsg: secret "image-registry-tls" not found Nov 03 14:11:02.139628 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:11:02.139512 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61060804-c519-490a-ac90-674792d50174-metrics-tls podName:61060804-c519-490a-ac90-674792d50174 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:10.13949678 +0000 UTC m=+45.755623380 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61060804-c519-490a-ac90-674792d50174-metrics-tls") pod "dns-default-2bk97" (UID: "61060804-c519-490a-ac90-674792d50174") : secret "dns-default-metrics-tls" not found Nov 03 14:11:02.139628 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:11:02.139528 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-registry-tls podName:b3260ef2-8ff4-4b70-a3a0-e5386a27d210 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:10.139519588 +0000 UTC m=+45.755646166 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-registry-tls") pod "image-registry-6d7c596f48-qrcsg" (UID: "b3260ef2-8ff4-4b70-a3a0-e5386a27d210") : secret "image-registry-tls" not found Nov 03 14:11:02.192555 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:02.192467 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pn8kf" event={"ID":"60d2d1a6-a89f-4a9a-acb9-e85ce2c8f6ce","Type":"ContainerStarted","Data":"d5ed3048d51d2d7d87085ac9f066e7d8e6b663be1405719040ec7c7ab734958a"} Nov 03 14:11:02.199696 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:02.198699 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fzk44-mlg6z" event={"ID":"7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d","Type":"ContainerStarted","Data":"a380b9bda8356290219aff6192ae489235555e3d14a3bc20f3c5b1e1cbe750a8"} Nov 03 14:11:02.199696 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:02.198756 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fzk44-mlg6z" event={"ID":"7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d","Type":"ContainerStarted","Data":"8d03b1024d60e73a8391f33068de1832e67c70d38283aa74f1366514ff8452c5"} Nov 03 14:11:02.201821 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:02.201794 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-c8hlq" event={"ID":"1da80d21-c62b-44ec-a3fe-64633b5de401","Type":"ContainerStarted","Data":"430a1e4878dba2ce0ac77cbc20ec0e6f83bd85eacc21aa2aeb477f543173caca"} Nov 03 14:11:02.242676 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:02.242615 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-fzk44-mlg6z" podStartSLOduration=2.242597149 podStartE2EDuration="2.242597149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:02.241321499 +0000 UTC m=+37.857448106" watchObservedRunningTime="2026-11-03 14:11:02.242597149 +0000 UTC m=+37.858723751" Nov 03 14:11:02.340775 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:02.340705 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sm2jg\" (UID: \"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee\") " pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:02.340947 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:02.340861 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sm2jg\" (UID: \"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee\") " pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:02.341010 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:11:02.340949 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 14:11:02.341010 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:11:02.340998 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-kube-rbac-proxy-cm podName:3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee nodeName:}" failed. No retries permitted until 2026-11-03 14:11:04.340986874 +0000 UTC m=+39.957113467 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sm2jg" (UID: "3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee") : configmap "kube-rbac-proxy" not found Nov 03 14:11:02.342170 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:11:02.342145 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 14:11:02.342276 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:11:02.342198 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-insights-runtime-extractor-tls podName:3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee nodeName:}" failed. No retries permitted until 2026-11-03 14:11:04.342184266 +0000 UTC m=+39.958310863 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sm2jg" (UID: "3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee") : secret "insights-runtime-extractor-tls" not found Nov 03 14:11:02.588870 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:02.588594 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-9vrkp"] Nov 03 14:11:02.600033 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:02.599933 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-9vrkp" Nov 03 14:11:02.618488 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:02.618319 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 14:11:02.618488 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:02.618449 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 14:11:02.618694 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:02.618534 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 14:11:02.618694 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:02.618622 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 14:11:02.618790 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:02.618714 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-hhz2n\"" Nov 03 14:11:02.619383 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:02.619359 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-9vrkp"] Nov 03 14:11:02.746017 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:02.745645 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/32d5aaf4-3430-4e5b-aca5-f0d720ce8657-signing-key\") pod \"service-ca-5fffc97bdf-9vrkp\" (UID: \"32d5aaf4-3430-4e5b-aca5-f0d720ce8657\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9vrkp" Nov 03 14:11:02.746017 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:02.745765 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvrrb\" (UniqueName: \"kubernetes.io/projected/32d5aaf4-3430-4e5b-aca5-f0d720ce8657-kube-api-access-nvrrb\") pod \"service-ca-5fffc97bdf-9vrkp\" (UID: \"32d5aaf4-3430-4e5b-aca5-f0d720ce8657\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9vrkp" Nov 03 14:11:02.746017 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:02.745854 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/32d5aaf4-3430-4e5b-aca5-f0d720ce8657-signing-cabundle\") pod \"service-ca-5fffc97bdf-9vrkp\" (UID: \"32d5aaf4-3430-4e5b-aca5-f0d720ce8657\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9vrkp" Nov 03 14:11:02.848773 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:02.848737 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/32d5aaf4-3430-4e5b-aca5-f0d720ce8657-signing-key\") pod \"service-ca-5fffc97bdf-9vrkp\" (UID: \"32d5aaf4-3430-4e5b-aca5-f0d720ce8657\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9vrkp" Nov 03 14:11:02.849006 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:02.848804 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvrrb\" (UniqueName: \"kubernetes.io/projected/32d5aaf4-3430-4e5b-aca5-f0d720ce8657-kube-api-access-nvrrb\") pod \"service-ca-5fffc97bdf-9vrkp\" (UID: \"32d5aaf4-3430-4e5b-aca5-f0d720ce8657\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9vrkp" Nov 03 14:11:02.849006 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:02.848887 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/32d5aaf4-3430-4e5b-aca5-f0d720ce8657-signing-cabundle\") pod \"service-ca-5fffc97bdf-9vrkp\" (UID: \"32d5aaf4-3430-4e5b-aca5-f0d720ce8657\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9vrkp" Nov 03 14:11:02.849694 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:02.849666 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/32d5aaf4-3430-4e5b-aca5-f0d720ce8657-signing-cabundle\") pod \"service-ca-5fffc97bdf-9vrkp\" (UID: \"32d5aaf4-3430-4e5b-aca5-f0d720ce8657\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9vrkp" Nov 03 14:11:02.854768 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:02.854713 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/32d5aaf4-3430-4e5b-aca5-f0d720ce8657-signing-key\") pod \"service-ca-5fffc97bdf-9vrkp\" (UID: \"32d5aaf4-3430-4e5b-aca5-f0d720ce8657\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9vrkp" Nov 03 14:11:02.872326 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:02.872294 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvrrb\" (UniqueName: \"kubernetes.io/projected/32d5aaf4-3430-4e5b-aca5-f0d720ce8657-kube-api-access-nvrrb\") pod \"service-ca-5fffc97bdf-9vrkp\" (UID: \"32d5aaf4-3430-4e5b-aca5-f0d720ce8657\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9vrkp" Nov 03 14:11:02.925388 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:02.925351 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-9vrkp" Nov 03 14:11:03.241418 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:03.241237 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-9vrkp"] Nov 03 14:11:03.283470 ip-10-0-143-171 kubenswrapper[2577]: W1103 14:11:03.283433 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32d5aaf4_3430_4e5b_aca5_f0d720ce8657.slice/crio-3f3072125365fd78ca387225052c6f2bdeaf31194b0d58f1497687c2422dee99 WatchSource:0}: Error finding container 3f3072125365fd78ca387225052c6f2bdeaf31194b0d58f1497687c2422dee99: Status 404 returned error can't find the container with id 3f3072125365fd78ca387225052c6f2bdeaf31194b0d58f1497687c2422dee99 Nov 03 14:11:04.211285 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:04.211247 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-c8hlq" event={"ID":"1da80d21-c62b-44ec-a3fe-64633b5de401","Type":"ContainerStarted","Data":"95e07ca5aa6508a27b989b49037efcd181ddf0ac2f77a7d8273135f4d9b8f925"} Nov 03 14:11:04.213004 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:04.212971 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-9vrkp" event={"ID":"32d5aaf4-3430-4e5b-aca5-f0d720ce8657","Type":"ContainerStarted","Data":"d6cba9c46295fb1aea608073eac9c26722deee635a68b012ce6ea2f42c01aeaf"} Nov 03 14:11:04.213133 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:04.213018 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-9vrkp" event={"ID":"32d5aaf4-3430-4e5b-aca5-f0d720ce8657","Type":"ContainerStarted","Data":"3f3072125365fd78ca387225052c6f2bdeaf31194b0d58f1497687c2422dee99"} Nov 03 14:11:04.214581 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:04.214553 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pn8kf" event={"ID":"60d2d1a6-a89f-4a9a-acb9-e85ce2c8f6ce","Type":"ContainerStarted","Data":"51e44052e427230b5072d0395525c6bc8da60a52e412cd2135518c533845abcd"} Nov 03 14:11:04.255938 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:04.255886 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-c8hlq" podStartSLOduration=2.331993888 podStartE2EDuration="4.255868363s" totalImagesPullingTime="1.923874475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:11:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:04.254667308 +0000 UTC m=+39.870793911" watchObservedRunningTime="2026-11-03 14:11:04.255868363 +0000 UTC m=+39.871994965" Nov 03 14:11:04.301797 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:04.299744 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pn8kf" podStartSLOduration=2.432652243 podStartE2EDuration="4.299726787s" totalImagesPullingTime="1.867074544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:11:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:04.296531485 +0000 UTC m=+39.912658088" watchObservedRunningTime="2026-11-03 14:11:04.299726787 +0000 UTC m=+39.915853387" Nov 03 14:11:04.366253 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:04.366213 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sm2jg\" (UID: \"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee\") " pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:04.366433 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:04.366355 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sm2jg\" (UID: \"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee\") " pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:04.366537 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:11:04.366520 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 14:11:04.366605 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:11:04.366585 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-insights-runtime-extractor-tls podName:3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee nodeName:}" failed. No retries permitted until 2026-11-03 14:11:08.36656787 +0000 UTC m=+43.982694468 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sm2jg" (UID: "3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee") : secret "insights-runtime-extractor-tls" not found Nov 03 14:11:04.367080 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:11:04.367059 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 14:11:04.367214 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:11:04.367113 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-kube-rbac-proxy-cm podName:3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee nodeName:}" failed. No retries permitted until 2026-11-03 14:11:08.367098795 +0000 UTC m=+43.983225392 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sm2jg" (UID: "3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee") : configmap "kube-rbac-proxy" not found Nov 03 14:11:04.610898 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:04.610860 2577 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" Nov 03 14:11:04.611078 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:04.610946 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" Nov 03 14:11:04.611448 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:04.611425 2577 scope.go:122] "RemoveContainer" containerID="d87a44a09d0ad13a61108d4537be53ab5164f5519ade849da1c1ba1e459d0490" Nov 03 14:11:04.611669 ip-10-0-143-171 kubenswrapper[2577]: E1103 14:11:04.611648 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5xzwg_openshift-console-operator(307e48f6-4a66-4c9c-a4ff-848cce9d3e0e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" podUID="307e48f6-4a66-4c9c-a4ff-848cce9d3e0e" Nov 03 14:11:05.252678 ip-10-0-143-171 kubenswrapper[2577]: I1103 14:11:05.252648 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q4gsb_e637e650-5fc6-4adb-a97a-47d924c30f97/dns-node-resolver/0.log" Nov 03 14:11:06.059578 ip-10-0-143-171 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 14:11:06.070199 ip-10-0-143-171 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 14:11:06.070363 ip-10-0-143-171 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 14:11:06.070411 ip-10-0-143-171 systemd[1]: kubelet.service: Consumed 3.136s CPU time. Nov 03 14:11:06.080406 ip-10-0-143-171 systemd[1]: Starting Kubernetes Kubelet... Nov 03 14:11:06.139199 ip-10-0-143-171 kubenswrapper[5060]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 14:11:06.139199 ip-10-0-143-171 kubenswrapper[5060]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 14:11:06.139199 ip-10-0-143-171 kubenswrapper[5060]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 14:11:06.139199 ip-10-0-143-171 kubenswrapper[5060]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 14:11:06.141589 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141504 5060 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 14:11:06.141589 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141583 5060 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 14:11:06.141589 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141588 5060 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 14:11:06.141589 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141592 5060 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 14:11:06.141589 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141596 5060 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 14:11:06.141933 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141600 5060 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 14:11:06.141933 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141604 5060 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 14:11:06.141933 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141608 5060 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 14:11:06.141933 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141612 5060 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 14:11:06.141933 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141617 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 14:11:06.141933 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141621 5060 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 14:11:06.141933 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141624 5060 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 14:11:06.141933 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141628 5060 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 14:11:06.141933 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141632 5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 14:11:06.141933 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141636 5060 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 14:11:06.141933 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141651 5060 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 14:11:06.141933 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141655 5060 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 14:11:06.141933 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141659 5060 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 14:11:06.141933 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141663 5060 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 14:11:06.141933 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141667 5060 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 14:11:06.141933 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141671 5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 14:11:06.141933 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141675 5060 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 14:11:06.141933 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141679 5060 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 14:11:06.141933 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141685 5060 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 14:11:06.141933 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141688 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 14:11:06.142819 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141693 5060 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 14:11:06.142819 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141697 5060 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 14:11:06.142819 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141701 5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 14:11:06.142819 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141705 5060 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 14:11:06.142819 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141709 5060 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 14:11:06.142819 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141713 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 14:11:06.142819 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141717 5060 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 14:11:06.142819 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141721 5060 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 14:11:06.142819 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141726 5060 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 14:11:06.142819 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141730 5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 14:11:06.142819 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141735 5060 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 14:11:06.142819 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141739 5060 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 14:11:06.142819 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141744 5060 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 14:11:06.142819 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141748 5060 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 14:11:06.142819 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141752 5060 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 14:11:06.142819 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141758 5060 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 14:11:06.142819 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141762 5060 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 14:11:06.142819 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141768 5060 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 14:11:06.142819 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141775 5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 14:11:06.142819 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141780 5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 14:11:06.143748 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141785 5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 14:11:06.143748 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141792 5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 14:11:06.143748 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141798 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 14:11:06.143748 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141803 5060 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 14:11:06.143748 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141808 5060 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 14:11:06.143748 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141812 5060 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 14:11:06.143748 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141819 5060 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 14:11:06.143748 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141826 5060 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 14:11:06.143748 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141848 5060 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 14:11:06.143748 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141854 5060 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 14:11:06.143748 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141858 5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 14:11:06.143748 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141862 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 14:11:06.143748 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141866 5060 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 14:11:06.143748 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141873 5060 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 14:11:06.143748 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141877 5060 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 14:11:06.143748 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141881 5060 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 14:11:06.143748 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141886 5060 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 14:11:06.143748 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141890 5060 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 14:11:06.143748 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141894 5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 14:11:06.144646 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141899 5060 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 14:11:06.144646 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141903 5060 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 14:11:06.144646 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141907 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 14:11:06.144646 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141912 5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 14:11:06.144646 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141916 5060 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 14:11:06.144646 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141921 5060 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 14:11:06.144646 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141926 5060 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 14:11:06.144646 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141930 5060 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 14:11:06.144646 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141934 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 14:11:06.144646 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141938 5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 14:11:06.144646 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141943 5060 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 14:11:06.144646 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141947 5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 14:11:06.144646 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141952 5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 14:11:06.144646 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141957 5060 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 14:11:06.144646 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141961 5060 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 14:11:06.144646 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141966 5060 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 14:11:06.144646 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141971 5060 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 14:11:06.144646 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141976 5060 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 14:11:06.144646 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141980 5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 14:11:06.144646 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141984 5060 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 14:11:06.145601 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141989 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 14:11:06.145601 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141993 5060 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 14:11:06.145601 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.141997 5060 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 14:11:06.145601 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.142002 5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 14:11:06.145601 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.142006 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 14:11:06.145601 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.142010 5060 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 14:11:06.145601 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.142015 5060 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 14:11:06.145601 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.142019 5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 14:11:06.145601 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.142023 5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 14:11:06.145601 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.142028 5060 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 14:11:06.145601 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.142034 5060 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 14:11:06.145601 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.142039 5060 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 14:11:06.145601 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.142043 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 14:11:06.145601 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.142047 5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 14:11:06.145601 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.142052 5060 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 14:11:06.145601 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.142056 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 14:11:06.145601 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.142060 5060 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 14:11:06.145601 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.142065 5060 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 14:11:06.145601 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.142069 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 14:11:06.146468 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.142073 5060 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 14:11:06.146468 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.142077 5060 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 14:11:06.146468 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.142081 5060 feature_gate.go:386] unrecognized feature gate: Example Nov 03 14:11:06.146468 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143405 5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 14:11:06.146468 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143416 5060 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 14:11:06.146468 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143421 5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 14:11:06.146468 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143426 5060 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 14:11:06.146468 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143431 5060 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 14:11:06.146468 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143436 5060 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 14:11:06.146468 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143441 5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 14:11:06.146468 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143446 5060 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 14:11:06.146468 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143450 5060 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 14:11:06.146468 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143455 5060 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 14:11:06.146468 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143459 5060 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 14:11:06.146468 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143464 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 14:11:06.146468 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143468 5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 14:11:06.146468 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143472 5060 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 14:11:06.146468 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143477 5060 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 14:11:06.146468 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143481 5060 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 14:11:06.146468 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143486 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 14:11:06.147441 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143490 5060 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 14:11:06.147441 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143494 5060 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 14:11:06.147441 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143499 5060 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 14:11:06.147441 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143503 5060 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 14:11:06.147441 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143508 5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 14:11:06.147441 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143512 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 14:11:06.147441 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143518 5060 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 14:11:06.147441 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143524 5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 14:11:06.147441 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143531 5060 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 14:11:06.147441 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143536 5060 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 14:11:06.147441 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143540 5060 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 14:11:06.147441 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143545 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 14:11:06.147441 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143549 5060 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 14:11:06.147441 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143553 5060 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 14:11:06.147441 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143558 5060 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 14:11:06.147441 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143562 5060 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 14:11:06.147441 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143566 5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 14:11:06.147441 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143570 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 14:11:06.147441 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143575 5060 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 14:11:06.148201 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143579 5060 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 14:11:06.148201 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143583 5060 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 14:11:06.148201 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143588 5060 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 14:11:06.148201 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143592 5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 14:11:06.148201 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143597 5060 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 14:11:06.148201 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143601 5060 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 14:11:06.148201 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143605 5060 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 14:11:06.148201 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143610 5060 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 14:11:06.148201 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143614 5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 14:11:06.148201 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143618 5060 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 14:11:06.148201 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143622 5060 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 14:11:06.148201 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143627 5060 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 14:11:06.148201 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143632 5060 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 14:11:06.148201 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143636 5060 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 14:11:06.148201 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143640 5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 14:11:06.148201 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143645 5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 14:11:06.148201 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143651 5060 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 14:11:06.148201 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143658 5060 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 14:11:06.148201 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143663 5060 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 14:11:06.148706 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143668 5060 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 14:11:06.148706 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143673 5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 14:11:06.148706 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143678 5060 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 14:11:06.148706 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143682 5060 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 14:11:06.148706 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143686 5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 14:11:06.148706 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143691 5060 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 14:11:06.148706 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143695 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 14:11:06.148706 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143699 5060 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 14:11:06.148706 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143704 5060 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 14:11:06.148706 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143708 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 14:11:06.148706 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143714 5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 14:11:06.148706 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143718 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 14:11:06.148706 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143722 5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 14:11:06.148706 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143727 5060 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 14:11:06.148706 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143731 5060 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 14:11:06.148706 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143736 5060 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 14:11:06.148706 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143740 5060 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 14:11:06.148706 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143745 5060 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 14:11:06.148706 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143750 5060 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 14:11:06.148706 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143754 5060 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 14:11:06.149269 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143759 5060 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 14:11:06.149269 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143763 5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 14:11:06.149269 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143767 5060 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 14:11:06.149269 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143772 5060 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 14:11:06.149269 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143777 5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 14:11:06.149269 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143782 5060 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 14:11:06.149269 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143786 5060 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 14:11:06.149269 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143790 5060 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 14:11:06.149269 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143794 5060 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 14:11:06.149269 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143799 5060 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 14:11:06.149269 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143803 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 14:11:06.149269 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143807 5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 14:11:06.149269 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143811 5060 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 14:11:06.149269 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143816 5060 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 14:11:06.149269 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143820 5060 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 14:11:06.149269 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143824 5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 14:11:06.149269 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143828 5060 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 14:11:06.149269 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143851 5060 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 14:11:06.149269 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143856 5060 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 14:11:06.149269 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143860 5060 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 14:11:06.149846 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143864 5060 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 14:11:06.149846 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143869 5060 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 14:11:06.149846 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143873 5060 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 14:11:06.149846 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143878 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 14:11:06.149846 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143882 5060 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 14:11:06.149846 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143898 5060 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 14:11:06.149846 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143903 5060 feature_gate.go:386] unrecognized feature gate: Example Nov 03 14:11:06.149846 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143907 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 14:11:06.149846 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143911 5060 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 14:11:06.149846 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143916 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 14:11:06.149846 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.143920 5060 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 14:11:06.149846 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144736 5060 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 14:11:06.149846 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144746 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 14:11:06.149846 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144750 5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 14:11:06.149846 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144755 5060 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 14:11:06.149846 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144760 5060 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 14:11:06.149846 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144764 5060 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 14:11:06.149846 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144769 5060 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 14:11:06.149846 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144774 5060 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 14:11:06.150401 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144778 5060 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 14:11:06.150401 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144783 5060 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 14:11:06.150401 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144787 5060 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 14:11:06.150401 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144792 5060 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 14:11:06.150401 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144796 5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 14:11:06.150401 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144800 5060 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 14:11:06.150401 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144804 5060 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 14:11:06.150401 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144809 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 14:11:06.150401 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144814 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 14:11:06.150401 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144818 5060 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 14:11:06.150401 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144823 5060 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 14:11:06.150401 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144845 5060 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 14:11:06.150401 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144852 5060 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 14:11:06.150401 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144857 5060 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 14:11:06.150401 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144862 5060 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 14:11:06.150401 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144867 5060 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 14:11:06.150401 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144871 5060 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 14:11:06.150401 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144877 5060 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 14:11:06.150401 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144884 5060 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 14:11:06.150951 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144889 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 14:11:06.150951 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144895 5060 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 14:11:06.150951 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144899 5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 14:11:06.150951 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144905 5060 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 14:11:06.150951 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144910 5060 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 14:11:06.150951 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144915 5060 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 14:11:06.150951 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144920 5060 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 14:11:06.150951 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144924 5060 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 14:11:06.150951 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144929 5060 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 14:11:06.150951 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144933 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 14:11:06.150951 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144938 5060 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 14:11:06.150951 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144942 5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 14:11:06.150951 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144946 5060 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 14:11:06.150951 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144951 5060 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 14:11:06.150951 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144955 5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 14:11:06.150951 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144961 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 14:11:06.150951 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144965 5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 14:11:06.150951 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144969 5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 14:11:06.150951 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144973 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 14:11:06.151635 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144978 5060 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 14:11:06.151635 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144982 5060 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 14:11:06.151635 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144987 5060 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 14:11:06.151635 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144991 5060 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 14:11:06.151635 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.144999 5060 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 14:11:06.151635 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145003 5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 14:11:06.151635 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145008 5060 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 14:11:06.151635 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145012 5060 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 14:11:06.151635 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145016 5060 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 14:11:06.151635 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145021 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 14:11:06.151635 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145025 5060 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 14:11:06.151635 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145029 5060 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 14:11:06.151635 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145034 5060 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 14:11:06.151635 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145038 5060 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 14:11:06.151635 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145042 5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 14:11:06.151635 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145046 5060 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 14:11:06.151635 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145050 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 14:11:06.151635 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145054 5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 14:11:06.151635 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145058 5060 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 14:11:06.151635 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145063 5060 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 14:11:06.152249 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145068 5060 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 14:11:06.152249 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145072 5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 14:11:06.152249 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145076 5060 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 14:11:06.152249 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145080 5060 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 14:11:06.152249 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145084 5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 14:11:06.152249 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145088 5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 14:11:06.152249 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145096 5060 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 14:11:06.152249 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145100 5060 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 14:11:06.152249 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145104 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 14:11:06.152249 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145108 5060 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 14:11:06.152249 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145112 5060 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 14:11:06.152249 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145117 5060 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 14:11:06.152249 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145121 5060 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 14:11:06.152249 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145126 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 14:11:06.152249 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145130 5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 14:11:06.152249 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145134 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 14:11:06.152249 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145140 5060 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 14:11:06.152249 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145144 5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 14:11:06.152249 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145149 5060 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 14:11:06.152734 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145153 5060 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 14:11:06.152734 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145157 5060 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 14:11:06.152734 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145161 5060 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 14:11:06.152734 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145166 5060 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 14:11:06.152734 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145170 5060 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 14:11:06.152734 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145174 5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 14:11:06.152734 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145178 5060 feature_gate.go:386] unrecognized feature gate: Example Nov 03 14:11:06.152734 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145183 5060 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 14:11:06.152734 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145187 5060 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 14:11:06.152734 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145191 5060 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 14:11:06.152734 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145195 5060 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 14:11:06.152734 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145200 5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 14:11:06.152734 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145206 5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 14:11:06.152734 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145211 5060 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 14:11:06.152734 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145215 5060 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 14:11:06.152734 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145219 5060 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 14:11:06.152734 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145224 5060 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 14:11:06.152734 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145229 5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 14:11:06.152734 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145233 5060 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 14:11:06.152734 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145239 5060 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 14:11:06.152734 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.145243 5060 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 14:11:06.153276 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145377 5060 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 14:11:06.153276 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145393 5060 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 14:11:06.153276 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145402 5060 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 14:11:06.153276 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145409 5060 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 14:11:06.153276 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145416 5060 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 14:11:06.153276 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145422 5060 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 14:11:06.153276 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145429 5060 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 14:11:06.153276 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145436 5060 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 14:11:06.153276 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145443 5060 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 14:11:06.153276 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145448 5060 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 14:11:06.153276 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145454 5060 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 14:11:06.153276 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145459 5060 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 14:11:06.153276 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145464 5060 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 14:11:06.153276 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145469 5060 flags.go:64] FLAG: --cgroup-root="" Nov 03 14:11:06.153276 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145473 5060 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 14:11:06.153276 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145479 5060 flags.go:64] FLAG: --client-ca-file="" Nov 03 14:11:06.153276 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145483 5060 flags.go:64] FLAG: --cloud-provider="external" Nov 03 14:11:06.153276 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145488 5060 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 14:11:06.153276 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145495 5060 flags.go:64] FLAG: --cluster-domain="" Nov 03 14:11:06.153276 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145500 5060 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 14:11:06.153276 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145505 5060 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 14:11:06.153276 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145510 5060 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 14:11:06.153276 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145516 5060 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 14:11:06.153276 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145522 5060 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 14:11:06.153886 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145527 5060 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 14:11:06.153886 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145533 5060 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 14:11:06.153886 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145538 5060 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 14:11:06.153886 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145543 5060 flags.go:64] FLAG: --contention-profiling="false" Nov 03 14:11:06.153886 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145548 5060 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 14:11:06.153886 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145553 5060 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 14:11:06.153886 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145561 5060 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 14:11:06.153886 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145566 5060 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 14:11:06.153886 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145573 5060 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 14:11:06.153886 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145578 5060 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 14:11:06.153886 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145583 5060 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 14:11:06.153886 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145588 5060 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 14:11:06.153886 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145593 5060 flags.go:64] FLAG: --enable-server="true" Nov 03 14:11:06.153886 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145597 5060 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 14:11:06.153886 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145603 5060 flags.go:64] FLAG: --event-burst="100" Nov 03 14:11:06.153886 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145607 5060 flags.go:64] FLAG: --event-qps="50" Nov 03 14:11:06.153886 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145614 5060 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 14:11:06.153886 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145642 5060 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 14:11:06.153886 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145650 5060 flags.go:64] FLAG: --eviction-hard="" Nov 03 14:11:06.153886 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145657 5060 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 14:11:06.153886 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145662 5060 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 14:11:06.153886 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145667 5060 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 14:11:06.153886 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145672 5060 flags.go:64] FLAG: --eviction-soft="" Nov 03 14:11:06.153886 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145677 5060 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 14:11:06.153886 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145682 5060 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 14:11:06.154510 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145688 5060 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 14:11:06.154510 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145692 5060 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 14:11:06.154510 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145697 5060 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 14:11:06.154510 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145702 5060 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 14:11:06.154510 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145707 5060 flags.go:64] FLAG: --feature-gates="" Nov 03 14:11:06.154510 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145713 5060 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 14:11:06.154510 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145719 5060 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 14:11:06.154510 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145724 5060 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 14:11:06.154510 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145729 5060 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 14:11:06.154510 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145735 5060 flags.go:64] FLAG: --healthz-port="10248" Nov 03 14:11:06.154510 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145740 5060 flags.go:64] FLAG: --help="false" Nov 03 14:11:06.154510 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145745 5060 flags.go:64] FLAG: --hostname-override="ip-10-0-143-171.ec2.internal" Nov 03 14:11:06.154510 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145750 5060 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 14:11:06.154510 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145757 5060 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 14:11:06.154510 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145762 5060 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 14:11:06.154510 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145767 5060 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 14:11:06.154510 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145773 5060 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 14:11:06.154510 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145778 5060 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 14:11:06.154510 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145783 5060 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 14:11:06.154510 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145787 5060 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 14:11:06.154510 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145792 5060 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 14:11:06.154510 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145797 5060 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 14:11:06.154510 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145802 5060 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 14:11:06.154510 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145809 5060 flags.go:64] FLAG: --kube-reserved="" Nov 03 14:11:06.155124 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145814 5060 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 14:11:06.155124 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145818 5060 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 14:11:06.155124 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145824 5060 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 14:11:06.155124 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145828 5060 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 14:11:06.155124 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145850 5060 flags.go:64] FLAG: --lock-file="" Nov 03 14:11:06.155124 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145855 5060 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 14:11:06.155124 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145860 5060 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 14:11:06.155124 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145865 5060 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 14:11:06.155124 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145874 5060 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 14:11:06.155124 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145879 5060 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 14:11:06.155124 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145883 5060 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 14:11:06.155124 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145889 5060 flags.go:64] FLAG: --logging-format="text" Nov 03 14:11:06.155124 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145894 5060 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 14:11:06.155124 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145900 5060 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 14:11:06.155124 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145904 5060 flags.go:64] FLAG: --manifest-url="" Nov 03 14:11:06.155124 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145909 5060 flags.go:64] FLAG: --manifest-url-header="" Nov 03 14:11:06.155124 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145916 5060 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 14:11:06.155124 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145921 5060 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 14:11:06.155124 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145928 5060 flags.go:64] FLAG: --max-pods="110" Nov 03 14:11:06.155124 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145934 5060 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 14:11:06.155124 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145939 5060 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 14:11:06.155124 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145945 5060 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 14:11:06.155124 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145950 5060 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 14:11:06.155124 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145955 5060 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 14:11:06.155124 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145959 5060 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 14:11:06.155742 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145965 5060 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 14:11:06.155742 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145976 5060 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 14:11:06.155742 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145981 5060 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 14:11:06.155742 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145986 5060 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 14:11:06.155742 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145991 5060 flags.go:64] FLAG: --pod-cidr="" Nov 03 14:11:06.155742 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.145996 5060 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 14:11:06.155742 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146002 5060 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 14:11:06.155742 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146007 5060 flags.go:64] FLAG: --pods-per-core="0" Nov 03 14:11:06.155742 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146011 5060 flags.go:64] FLAG: --port="10250" Nov 03 14:11:06.155742 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146017 5060 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 14:11:06.155742 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146021 5060 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c10c1b9db03ccfbc" Nov 03 14:11:06.155742 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146027 5060 flags.go:64] FLAG: --qos-reserved="" Nov 03 14:11:06.155742 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146032 5060 flags.go:64] FLAG: --read-only-port="10255" Nov 03 14:11:06.155742 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146036 5060 flags.go:64] FLAG: --register-node="true" Nov 03 14:11:06.155742 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146041 5060 flags.go:64] FLAG: --register-with-taints="" Nov 03 14:11:06.155742 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146047 5060 flags.go:64] FLAG: --registry-burst="10" Nov 03 14:11:06.155742 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146052 5060 flags.go:64] FLAG: --registry-qps="5" Nov 03 14:11:06.155742 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146056 5060 flags.go:64] FLAG: --reserved-cpus="" Nov 03 14:11:06.155742 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146061 5060 flags.go:64] FLAG: --reserved-memory="" Nov 03 14:11:06.155742 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146067 5060 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 14:11:06.155742 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146072 5060 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 14:11:06.155742 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146077 5060 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 14:11:06.155742 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146082 5060 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 14:11:06.155742 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146086 5060 flags.go:64] FLAG: --runonce="false" Nov 03 14:11:06.155742 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146091 5060 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 14:11:06.155742 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146096 5060 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 14:11:06.156483 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146102 5060 flags.go:64] FLAG: --seccomp-default="false" Nov 03 14:11:06.156483 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146107 5060 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 14:11:06.156483 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146114 5060 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 14:11:06.156483 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146119 5060 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 14:11:06.156483 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146124 5060 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 14:11:06.156483 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146129 5060 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 14:11:06.156483 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146134 5060 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 14:11:06.156483 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146138 5060 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 14:11:06.156483 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146143 5060 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 14:11:06.156483 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146148 5060 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 14:11:06.156483 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146153 5060 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 14:11:06.156483 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146158 5060 flags.go:64] FLAG: --system-cgroups="" Nov 03 14:11:06.156483 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146165 5060 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 14:11:06.156483 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146173 5060 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 14:11:06.156483 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146178 5060 flags.go:64] FLAG: --tls-cert-file="" Nov 03 14:11:06.156483 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146182 5060 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 14:11:06.156483 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146189 5060 flags.go:64] FLAG: --tls-min-version="" Nov 03 14:11:06.156483 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146194 5060 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 14:11:06.156483 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146198 5060 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 14:11:06.156483 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146203 5060 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 14:11:06.156483 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146208 5060 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 14:11:06.156483 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146213 5060 flags.go:64] FLAG: --v="2" Nov 03 14:11:06.156483 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146219 5060 flags.go:64] FLAG: --version="false" Nov 03 14:11:06.156483 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146225 5060 flags.go:64] FLAG: --vmodule="" Nov 03 14:11:06.156483 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146239 5060 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 14:11:06.157113 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.146245 5060 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 14:11:06.157113 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146454 5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 14:11:06.157113 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146462 5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 14:11:06.157113 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146466 5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 14:11:06.157113 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146470 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 14:11:06.157113 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146474 5060 feature_gate.go:386] unrecognized feature gate: Example Nov 03 14:11:06.157113 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146478 5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 14:11:06.157113 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146483 5060 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 14:11:06.157113 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146487 5060 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 14:11:06.157113 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146495 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 14:11:06.157113 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146500 5060 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 14:11:06.157113 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146504 5060 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 14:11:06.157113 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146508 5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 14:11:06.157113 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146513 5060 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 14:11:06.157113 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146517 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 14:11:06.157113 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146521 5060 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 14:11:06.157113 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146526 5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 14:11:06.157113 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146531 5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 14:11:06.157113 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146535 5060 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 14:11:06.157113 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146541 5060 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 14:11:06.157610 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146546 5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 14:11:06.157610 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146550 5060 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 14:11:06.157610 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146554 5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 14:11:06.157610 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146559 5060 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 14:11:06.157610 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146563 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 14:11:06.157610 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146568 5060 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 14:11:06.157610 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146572 5060 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 14:11:06.157610 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146577 5060 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 14:11:06.157610 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146581 5060 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 14:11:06.157610 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146585 5060 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 14:11:06.157610 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146589 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 14:11:06.157610 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146594 5060 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 14:11:06.157610 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146598 5060 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 14:11:06.157610 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146604 5060 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 14:11:06.157610 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146610 5060 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 14:11:06.157610 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146615 5060 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 14:11:06.157610 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146620 5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 14:11:06.157610 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146625 5060 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 14:11:06.157610 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146629 5060 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 14:11:06.157610 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146633 5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 14:11:06.158118 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146638 5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 14:11:06.158118 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146645 5060 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 14:11:06.158118 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146649 5060 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 14:11:06.158118 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146654 5060 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 14:11:06.158118 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146661 5060 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 14:11:06.158118 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146667 5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 14:11:06.158118 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146671 5060 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 14:11:06.158118 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146676 5060 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 14:11:06.158118 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146680 5060 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 14:11:06.158118 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146685 5060 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 14:11:06.158118 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146689 5060 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 14:11:06.158118 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146696 5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 14:11:06.158118 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146700 5060 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 14:11:06.158118 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146705 5060 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 14:11:06.158118 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146709 5060 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 14:11:06.158118 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146713 5060 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 14:11:06.158118 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146717 5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 14:11:06.158118 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146721 5060 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 14:11:06.158118 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146726 5060 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 14:11:06.158625 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146730 5060 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 14:11:06.158625 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146735 5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 14:11:06.158625 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146740 5060 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 14:11:06.158625 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146744 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 14:11:06.158625 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146748 5060 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 14:11:06.158625 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146752 5060 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 14:11:06.158625 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146756 5060 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 14:11:06.158625 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146761 5060 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 14:11:06.158625 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146765 5060 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 14:11:06.158625 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146769 5060 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 14:11:06.158625 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146773 5060 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 14:11:06.158625 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146777 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 14:11:06.158625 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146782 5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 14:11:06.158625 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146787 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 14:11:06.158625 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146793 5060 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 14:11:06.158625 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146798 5060 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 14:11:06.158625 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146802 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 14:11:06.158625 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146807 5060 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 14:11:06.158625 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146811 5060 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 14:11:06.159367 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146816 5060 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 14:11:06.159367 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146820 5060 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 14:11:06.159367 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146825 5060 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 14:11:06.159367 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146829 5060 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 14:11:06.159367 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146850 5060 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 14:11:06.159367 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146856 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 14:11:06.159367 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146861 5060 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 14:11:06.159367 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146866 5060 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 14:11:06.159367 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146870 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 14:11:06.159367 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146874 5060 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 14:11:06.159367 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146879 5060 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 14:11:06.159367 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146883 5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 14:11:06.159367 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146888 5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 14:11:06.159367 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146892 5060 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 14:11:06.159367 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146896 5060 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 14:11:06.159367 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146901 5060 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 14:11:06.159367 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146905 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 14:11:06.159367 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146909 5060 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 14:11:06.159367 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146913 5060 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 14:11:06.159367 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146917 5060 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 14:11:06.159910 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146921 5060 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 14:11:06.159910 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146926 5060 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 14:11:06.159910 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146930 5060 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 14:11:06.159910 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146934 5060 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 14:11:06.159910 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146939 5060 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 14:11:06.159910 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146943 5060 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 14:11:06.159910 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146947 5060 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 14:11:06.159910 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146954 5060 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 14:11:06.159910 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.146958 5060 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 14:11:06.159910 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.147002 5060 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 14:11:06.159910 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147058 5060 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 14:11:06.159910 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147065 5060 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 14:11:06.159910 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147071 5060 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 14:11:06.159910 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147076 5060 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 14:11:06.159910 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147080 5060 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 14:11:06.159910 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147085 5060 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 14:11:06.160314 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147089 5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 14:11:06.160314 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147096 5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 14:11:06.160314 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147100 5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 14:11:06.160314 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147104 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 14:11:06.160314 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147109 5060 feature_gate.go:386] unrecognized feature gate: Example Nov 03 14:11:06.160314 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147113 5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 14:11:06.160314 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147117 5060 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 14:11:06.160314 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147121 5060 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 14:11:06.160314 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147126 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 14:11:06.160314 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147130 5060 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 14:11:06.160314 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147135 5060 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 14:11:06.160314 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147139 5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 14:11:06.160314 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147143 5060 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 14:11:06.160314 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147147 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 14:11:06.160314 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147152 5060 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 14:11:06.160314 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147156 5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 14:11:06.160314 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147160 5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 14:11:06.160314 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147164 5060 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 14:11:06.160314 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147169 5060 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 14:11:06.160823 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147173 5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 14:11:06.160823 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147177 5060 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 14:11:06.160823 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147182 5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 14:11:06.160823 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147186 5060 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 14:11:06.160823 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147190 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 14:11:06.160823 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147197 5060 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 14:11:06.160823 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147201 5060 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 14:11:06.160823 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147206 5060 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 14:11:06.160823 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147210 5060 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 14:11:06.160823 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147214 5060 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 14:11:06.160823 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147219 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 14:11:06.160823 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147224 5060 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 14:11:06.160823 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147228 5060 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 14:11:06.160823 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147234 5060 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 14:11:06.160823 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147246 5060 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 14:11:06.160823 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147251 5060 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 14:11:06.160823 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147256 5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 14:11:06.160823 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147260 5060 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 14:11:06.160823 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147264 5060 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 14:11:06.160823 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147268 5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 14:11:06.161477 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147273 5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 14:11:06.161477 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147277 5060 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 14:11:06.161477 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147281 5060 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 14:11:06.161477 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147286 5060 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 14:11:06.161477 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147293 5060 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 14:11:06.161477 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147298 5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 14:11:06.161477 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147303 5060 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 14:11:06.161477 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147307 5060 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 14:11:06.161477 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147311 5060 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 14:11:06.161477 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147316 5060 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 14:11:06.161477 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147320 5060 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 14:11:06.161477 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147325 5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 14:11:06.161477 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147329 5060 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 14:11:06.161477 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147333 5060 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 14:11:06.161477 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147337 5060 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 14:11:06.161477 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147341 5060 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 14:11:06.161477 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147345 5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 14:11:06.161477 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147352 5060 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 14:11:06.161477 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147356 5060 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 14:11:06.162065 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147361 5060 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 14:11:06.162065 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147366 5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 14:11:06.162065 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147371 5060 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 14:11:06.162065 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147375 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 14:11:06.162065 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147379 5060 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 14:11:06.162065 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147383 5060 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 14:11:06.162065 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147388 5060 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 14:11:06.162065 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147393 5060 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 14:11:06.162065 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147397 5060 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 14:11:06.162065 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147402 5060 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 14:11:06.162065 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147406 5060 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 14:11:06.162065 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147411 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 14:11:06.162065 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147415 5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 14:11:06.162065 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147419 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 14:11:06.162065 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147423 5060 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 14:11:06.162065 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147429 5060 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 14:11:06.162065 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147433 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 14:11:06.162065 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147437 5060 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 14:11:06.162065 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147441 5060 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 14:11:06.162566 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147445 5060 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 14:11:06.162566 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147449 5060 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 14:11:06.162566 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147453 5060 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 14:11:06.162566 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147457 5060 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 14:11:06.162566 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147461 5060 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 14:11:06.162566 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147465 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 14:11:06.162566 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147470 5060 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 14:11:06.162566 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147475 5060 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 14:11:06.162566 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147479 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 14:11:06.162566 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147484 5060 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 14:11:06.162566 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147488 5060 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 14:11:06.162566 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147493 5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 14:11:06.162566 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147497 5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 14:11:06.162566 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147501 5060 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 14:11:06.162566 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147505 5060 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 14:11:06.162566 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147510 5060 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 14:11:06.162566 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147514 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 14:11:06.162566 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147519 5060 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 14:11:06.162566 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147523 5060 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 14:11:06.162566 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147528 5060 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 14:11:06.163068 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147534 5060 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 14:11:06.163068 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147538 5060 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 14:11:06.163068 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.147542 5060 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 14:11:06.163068 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.153362 5060 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 14:11:06.163068 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.153380 5060 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 14:11:06.163068 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.153398 5060 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 14:11:06.163068 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.153403 5060 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 14:11:06.163068 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153464 5060 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 14:11:06.163068 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153469 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 14:11:06.163068 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153473 5060 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 14:11:06.163068 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153476 5060 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 14:11:06.163068 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153479 5060 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 14:11:06.163068 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153482 5060 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 14:11:06.163068 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153486 5060 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 14:11:06.163068 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153490 5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 14:11:06.163068 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153493 5060 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 14:11:06.163068 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153497 5060 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 14:11:06.163068 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153499 5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 14:11:06.163068 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153502 5060 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 14:11:06.163627 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153505 5060 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 14:11:06.163627 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153508 5060 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 14:11:06.163627 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153511 5060 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 14:11:06.163627 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153513 5060 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 14:11:06.163627 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153516 5060 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 14:11:06.163627 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153519 5060 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 14:11:06.163627 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153521 5060 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 14:11:06.163627 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153524 5060 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 14:11:06.163627 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153526 5060 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 14:11:06.163627 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153529 5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 14:11:06.163627 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153532 5060 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 14:11:06.163627 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153535 5060 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 14:11:06.163627 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153537 5060 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 14:11:06.163627 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153540 5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 14:11:06.163627 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153543 5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 14:11:06.163627 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153546 5060 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 14:11:06.163627 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153549 5060 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 14:11:06.163627 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153552 5060 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 14:11:06.163627 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153555 5060 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 14:11:06.163627 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153557 5060 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 14:11:06.164145 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153560 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 14:11:06.164145 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153563 5060 feature_gate.go:386] unrecognized feature gate: Example Nov 03 14:11:06.164145 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153565 5060 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 14:11:06.164145 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153568 5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 14:11:06.164145 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153570 5060 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 14:11:06.164145 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153573 5060 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 14:11:06.164145 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153576 5060 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 14:11:06.164145 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153578 5060 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 14:11:06.164145 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153581 5060 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 14:11:06.164145 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153584 5060 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 14:11:06.164145 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153587 5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 14:11:06.164145 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153589 5060 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 14:11:06.164145 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153592 5060 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 14:11:06.164145 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153594 5060 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 14:11:06.164145 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153597 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 14:11:06.164145 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153600 5060 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 14:11:06.164145 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153603 5060 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 14:11:06.164145 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153606 5060 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 14:11:06.164145 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153608 5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 14:11:06.164145 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153611 5060 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 14:11:06.164739 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153614 5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 14:11:06.164739 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153616 5060 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 14:11:06.164739 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153619 5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 14:11:06.164739 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153622 5060 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 14:11:06.164739 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153624 5060 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 14:11:06.164739 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153627 5060 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 14:11:06.164739 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153629 5060 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 14:11:06.164739 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153633 5060 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 14:11:06.164739 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153637 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 14:11:06.164739 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153640 5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 14:11:06.164739 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153643 5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 14:11:06.164739 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153646 5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 14:11:06.164739 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153649 5060 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 14:11:06.164739 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153652 5060 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 14:11:06.164739 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153654 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 14:11:06.164739 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153657 5060 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 14:11:06.164739 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153660 5060 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 14:11:06.164739 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153663 5060 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 14:11:06.164739 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153665 5060 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 14:11:06.165343 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153668 5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 14:11:06.165343 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153670 5060 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 14:11:06.165343 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153673 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 14:11:06.165343 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153676 5060 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 14:11:06.165343 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153678 5060 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 14:11:06.165343 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153681 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 14:11:06.165343 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153684 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 14:11:06.165343 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153686 5060 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 14:11:06.165343 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153689 5060 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 14:11:06.165343 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153692 5060 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 14:11:06.165343 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153695 5060 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 14:11:06.165343 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153697 5060 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 14:11:06.165343 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153700 5060 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 14:11:06.165343 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153702 5060 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 14:11:06.165343 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153705 5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 14:11:06.165343 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153708 5060 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 14:11:06.165343 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153710 5060 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 14:11:06.165343 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153713 5060 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 14:11:06.165343 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153716 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 14:11:06.165343 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153718 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 14:11:06.166186 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153721 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 14:11:06.166186 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153723 5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 14:11:06.166186 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153726 5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 14:11:06.166186 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153728 5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 14:11:06.166186 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153731 5060 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 14:11:06.166186 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153734 5060 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 14:11:06.166186 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153736 5060 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 14:11:06.166186 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153739 5060 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 14:11:06.166186 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153742 5060 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 14:11:06.166186 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153744 5060 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 14:11:06.166186 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153747 5060 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 14:11:06.166186 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153749 5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 14:11:06.166186 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153752 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 14:11:06.166186 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153755 5060 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 14:11:06.166186 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153757 5060 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 14:11:06.166186 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.153784 5060 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 14:11:06.166729 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153963 5060 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 14:11:06.166729 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153968 5060 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 14:11:06.166729 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153971 5060 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 14:11:06.166729 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153975 5060 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 14:11:06.166729 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153980 5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 14:11:06.166729 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153984 5060 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 14:11:06.166729 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153987 5060 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 14:11:06.166729 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153990 5060 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 14:11:06.166729 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153992 5060 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 14:11:06.166729 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153995 5060 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 14:11:06.166729 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.153998 5060 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 14:11:06.166729 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154000 5060 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 14:11:06.166729 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154003 5060 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 14:11:06.166729 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154005 5060 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 14:11:06.166729 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154008 5060 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 14:11:06.166729 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154011 5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 14:11:06.166729 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154013 5060 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 14:11:06.166729 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154016 5060 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 14:11:06.166729 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154018 5060 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 14:11:06.167336 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154021 5060 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 14:11:06.167336 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154024 5060 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 14:11:06.167336 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154027 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 14:11:06.167336 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154030 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 14:11:06.167336 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154033 5060 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 14:11:06.167336 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154036 5060 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 14:11:06.167336 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154039 5060 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 14:11:06.167336 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154041 5060 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 14:11:06.167336 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154044 5060 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 14:11:06.167336 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154046 5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 14:11:06.167336 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154049 5060 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 14:11:06.167336 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154051 5060 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 14:11:06.167336 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154054 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 14:11:06.167336 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154056 5060 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 14:11:06.167336 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154059 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 14:11:06.167336 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154061 5060 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 14:11:06.167336 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154064 5060 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 14:11:06.167336 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154066 5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 14:11:06.167336 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154069 5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 14:11:06.167336 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154072 5060 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 14:11:06.168007 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154075 5060 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 14:11:06.168007 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154077 5060 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 14:11:06.168007 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154080 5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 14:11:06.168007 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154082 5060 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 14:11:06.168007 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154085 5060 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 14:11:06.168007 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154087 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 14:11:06.168007 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154090 5060 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 14:11:06.168007 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154093 5060 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 14:11:06.168007 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154095 5060 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 14:11:06.168007 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154098 5060 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 14:11:06.168007 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154100 5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 14:11:06.168007 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154103 5060 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 14:11:06.168007 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154105 5060 feature_gate.go:386] unrecognized feature gate: Example Nov 03 14:11:06.168007 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154108 5060 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 14:11:06.168007 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154111 5060 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 14:11:06.168007 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154113 5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 14:11:06.168007 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154116 5060 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 14:11:06.168007 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154119 5060 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 14:11:06.168007 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154121 5060 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 14:11:06.168007 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154124 5060 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 14:11:06.168805 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154126 5060 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 14:11:06.168805 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154129 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 14:11:06.168805 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154133 5060 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 14:11:06.168805 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154136 5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 14:11:06.168805 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154140 5060 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 14:11:06.168805 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154143 5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 14:11:06.168805 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154145 5060 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 14:11:06.168805 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154148 5060 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 14:11:06.168805 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154150 5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 14:11:06.168805 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154153 5060 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 14:11:06.168805 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154155 5060 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 14:11:06.168805 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154158 5060 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 14:11:06.168805 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154162 5060 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 14:11:06.168805 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154165 5060 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 14:11:06.168805 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154167 5060 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 14:11:06.168805 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154170 5060 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 14:11:06.168805 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154172 5060 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 14:11:06.168805 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154175 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 14:11:06.168805 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154177 5060 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 14:11:06.168805 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154180 5060 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 14:11:06.169463 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154183 5060 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 14:11:06.169463 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154186 5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 14:11:06.169463 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154188 5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 14:11:06.169463 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154191 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 14:11:06.169463 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154193 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 14:11:06.169463 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154196 5060 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 14:11:06.169463 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154198 5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 14:11:06.169463 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154201 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 14:11:06.169463 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154203 5060 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 14:11:06.169463 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154206 5060 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 14:11:06.169463 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154209 5060 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 14:11:06.169463 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154211 5060 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 14:11:06.169463 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154213 5060 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 14:11:06.169463 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154217 5060 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 14:11:06.169463 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154219 5060 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 14:11:06.169463 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154222 5060 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 14:11:06.169463 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154224 5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 14:11:06.169463 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154227 5060 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 14:11:06.169463 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154229 5060 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 14:11:06.170010 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154232 5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 14:11:06.170010 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154234 5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 14:11:06.170010 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154237 5060 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 14:11:06.170010 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154240 5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 14:11:06.170010 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154242 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 14:11:06.170010 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154245 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 14:11:06.170010 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154248 5060 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 14:11:06.170010 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154251 5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 14:11:06.170010 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.154275 5060 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 14:11:06.170010 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154308 5060 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 14:11:06.170010 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154312 5060 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 14:11:06.170010 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154316 5060 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 14:11:06.170010 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154319 5060 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 14:11:06.170010 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154322 5060 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 14:11:06.170010 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154325 5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 14:11:06.170010 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154327 5060 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 14:11:06.170427 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154330 5060 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 14:11:06.170427 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154332 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 14:11:06.170427 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154335 5060 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 14:11:06.170427 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154337 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 14:11:06.170427 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154340 5060 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 14:11:06.170427 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154342 5060 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 14:11:06.170427 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154345 5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 14:11:06.170427 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154348 5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 14:11:06.170427 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154350 5060 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 14:11:06.170427 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154353 5060 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 14:11:06.170427 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154356 5060 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 14:11:06.170427 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154358 5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 14:11:06.170427 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154361 5060 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 14:11:06.170427 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154364 5060 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 14:11:06.170427 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154366 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 14:11:06.170427 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154369 5060 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 14:11:06.170427 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154371 5060 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 14:11:06.170427 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154374 5060 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 14:11:06.170427 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154377 5060 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 14:11:06.170427 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154379 5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 14:11:06.170956 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154382 5060 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 14:11:06.170956 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154387 5060 feature_gate.go:386] unrecognized feature gate: Example Nov 03 14:11:06.170956 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154390 5060 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 14:11:06.170956 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154393 5060 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 14:11:06.170956 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154395 5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 14:11:06.170956 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154398 5060 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 14:11:06.170956 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154400 5060 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 14:11:06.170956 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154403 5060 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 14:11:06.170956 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154406 5060 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 14:11:06.170956 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154408 5060 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 14:11:06.170956 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154411 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 14:11:06.170956 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154415 5060 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 14:11:06.170956 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154419 5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 14:11:06.170956 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154422 5060 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 14:11:06.170956 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154425 5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 14:11:06.170956 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154428 5060 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 14:11:06.170956 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154430 5060 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 14:11:06.170956 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154433 5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 14:11:06.170956 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154436 5060 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 14:11:06.170956 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154438 5060 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 14:11:06.171441 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154441 5060 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 14:11:06.171441 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154444 5060 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 14:11:06.171441 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154446 5060 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 14:11:06.171441 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154449 5060 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 14:11:06.171441 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154451 5060 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 14:11:06.171441 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154454 5060 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 14:11:06.171441 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154456 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 14:11:06.171441 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154459 5060 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 14:11:06.171441 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154461 5060 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 14:11:06.171441 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154464 5060 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 14:11:06.171441 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154467 5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 14:11:06.171441 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154469 5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 14:11:06.171441 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154472 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 14:11:06.171441 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154477 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 14:11:06.171441 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154479 5060 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 14:11:06.171441 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154482 5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 14:11:06.171441 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154485 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 14:11:06.171441 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154488 5060 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 14:11:06.171441 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154491 5060 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 14:11:06.171931 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154493 5060 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 14:11:06.171931 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154496 5060 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 14:11:06.171931 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154498 5060 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 14:11:06.171931 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154501 5060 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 14:11:06.171931 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154504 5060 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 14:11:06.171931 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154507 5060 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 14:11:06.171931 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154509 5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 14:11:06.171931 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154512 5060 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 14:11:06.171931 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154515 5060 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 14:11:06.171931 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154518 5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 14:11:06.171931 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154520 5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 14:11:06.171931 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154523 5060 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 14:11:06.171931 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154526 5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 14:11:06.171931 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154529 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 14:11:06.171931 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154531 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 14:11:06.171931 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154534 5060 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 14:11:06.171931 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154537 5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 14:11:06.171931 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154540 5060 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 14:11:06.171931 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154542 5060 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 14:11:06.171931 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154545 5060 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 14:11:06.172451 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154549 5060 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 14:11:06.172451 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154552 5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 14:11:06.172451 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154556 5060 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 14:11:06.172451 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154559 5060 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 14:11:06.172451 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154562 5060 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 14:11:06.172451 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154565 5060 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 14:11:06.172451 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154569 5060 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 14:11:06.172451 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154571 5060 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 14:11:06.172451 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154574 5060 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 14:11:06.172451 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154577 5060 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 14:11:06.172451 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154580 5060 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 14:11:06.172451 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154583 5060 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 14:11:06.172451 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154585 5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 14:11:06.172451 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154588 5060 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 14:11:06.172451 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154590 5060 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 14:11:06.172451 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154593 5060 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 14:11:06.172451 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154595 5060 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 14:11:06.172451 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154598 5060 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 14:11:06.172451 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154600 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 14:11:06.172936 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:06.154604 5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 14:11:06.172936 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.154659 5060 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 14:11:06.172936 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.155639 5060 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 14:11:06.172936 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.155706 5060 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 14:11:06.172936 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.155980 5060 server.go:1014] "Starting client certificate rotation" Nov 03 14:11:06.172936 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.156119 5060 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 14:11:06.172936 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.156174 5060 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 14:05:24 +0000 UTC" deadline="2028-06-20 06:28:04.65509832 +0000 UTC" Nov 03 14:11:06.172936 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.156202 5060 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14272h16m58.498898024s" Nov 03 14:11:06.172936 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.156419 5060 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 14:11:06.172936 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.157696 5060 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 14:11:06.172936 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.159052 5060 log.go:25] "Validated CRI v1 runtime API" Nov 03 14:11:06.172936 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.160722 5060 log.go:25] "Validated CRI v1 image API" Nov 03 14:11:06.172936 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.161477 5060 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 14:11:06.172936 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.163416 5060 fs.go:135] Filesystem UUIDs: map[35141d3c-382c-4f7d-ac36-de190683c446:/dev/nvme0n1p3 37e8a46c-f295-4990-8032-e820db68c14f:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 03 14:11:06.173311 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.163442 5060 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/13785db1c6914236913af97f8c8f3777a7a9ebf03e55a39f25ff3e08eb9c0328/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/13785db1c6914236913af97f8c8f3777a7a9ebf03e55a39f25ff3e08eb9c0328/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/14d85150d7edbe1991a5ff99576317c8d72a74e40cd748fddc80df8fbf48a7c2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/14d85150d7edbe1991a5ff99576317c8d72a74e40cd748fddc80df8fbf48a7c2/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1da2eca1baf31392b9832da78c58b8a61a6e04b0b418d6920124b73c4c7d4b03/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1da2eca1baf31392b9832da78c58b8a61a6e04b0b418d6920124b73c4c7d4b03/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2867061ef7d1b1c91f970babfd5e957a2dc2ae00bff59a673a22086f0cf86e89/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2867061ef7d1b1c91f970babfd5e957a2dc2ae00bff59a673a22086f0cf86e89/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2ef0fa9513f22e1f5925f997bf3f1f4eeb9577ab139104228f84d2dd73133f20/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2ef0fa9513f22e1f5925f997bf3f1f4eeb9577ab139104228f84d2dd73133f20/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3f3072125365fd78ca387225052c6f2bdeaf31194b0d58f1497687c2422dee99/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3f3072125365fd78ca387225052c6f2bdeaf31194b0d58f1497687c2422dee99/userdata/shm major:0 minor:362 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/430a1e4878dba2ce0ac77cbc20ec0e6f83bd85eacc21aa2aeb477f543173caca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/430a1e4878dba2ce0ac77cbc20ec0e6f83bd85eacc21aa2aeb477f543173caca/userdata/shm major:0 minor:344 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6267a5ca51fba195deca8d7782843172823bedf31516805702604683e8d17646/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6267a5ca51fba195deca8d7782843172823bedf31516805702604683e8d17646/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7640e0f73cde1a06114a434eeb5ceb533b361171bd334ae3fff87ccbb55c03ff/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7640e0f73cde1a06114a434eeb5ceb533b361171bd334ae3fff87ccbb55c03ff/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8d03b1024d60e73a8391f33068de1832e67c70d38283aa74f1366514ff8452c5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8d03b1024d60e73a8391f33068de1832e67c70d38283aa74f1366514ff8452c5/userdata/shm major:0 minor:335 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9876113882c1611251a2fc38d5a386ff1f49e928e66cd1a51b21a3d5a08542ae/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9876113882c1611251a2fc38d5a386ff1f49e928e66cd1a51b21a3d5a08542ae/userdata/shm major:0 minor:293 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9c4f3c61cca7f03a33d5ad5e9a4a82825d334b9c05f2baebe6ab49f2dee6d18b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9c4f3c61cca7f03a33d5ad5e9a4a82825d334b9c05f2baebe6ab49f2dee6d18b/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a7a404869ab852ee9956a1299567c7d73017b8aa057e1718974ce544148abd5f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a7a404869ab852ee9956a1299567c7d73017b8aa057e1718974ce544148abd5f/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ae33d6534d669fa34314d880c5da457d96e430d11244450ad159fd464d82ea95/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ae33d6534d669fa34314d880c5da457d96e430d11244450ad159fd464d82ea95/userdata/shm major:0 minor:295 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b6867239ea6208777242d582c882a81cd26834e80a64ddff473a7fdc1e89e0fd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b6867239ea6208777242d582c882a81cd26834e80a64ddff473a7fdc1e89e0fd/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c4519af9ec99a107edf3cad6b845dcf115f26190a900862df518f5307e681b6a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c4519af9ec99a107edf3cad6b845dcf115f26190a900862df518f5307e681b6a/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ccae5bf3c9ab1e801ea85c8e8dea88e7011d7b98b5015e975f65b768ea2eb6c3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ccae5bf3c9ab1e801ea85c8e8dea88e7011d7b98b5015e975f65b768ea2eb6c3/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d5ed3048d51d2d7d87085ac9f066e7d8e6b663be1405719040ec7c7ab734958a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d5ed3048d51d2d7d87085ac9f066e7d8e6b663be1405719040ec7c7ab734958a/userdata/shm major:0 minor:346 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/db44b327584d75050b083495c3de9df2448c9f260f6100e73077383104187e6b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/db44b327584d75050b083495c3de9df2448c9f260f6100e73077383104187e6b/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dcb4a147277102c0ee92ed924dc29e6304010e9c9c62749f58c0cc45c6f40640/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dcb4a147277102c0ee92ed924dc29e6304010e9c9c62749f58c0cc45c6f40640/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dd6604a80c529a9ea0dfffe1743aecf939cd6a48243ad6ffa3691bb8cfe13087/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dd6604a80c529a9ea0dfffe1743aecf939cd6a48243ad6ffa3691bb8cfe13087/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ddafe78a5f8133c3e0127472345a985b2211323b1eefc6f97145d0d119c24cd2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ddafe78a5f8133c3e0127472345a985b2211323b1eefc6f97145d0d119c24cd2/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e16a0074955a8d79bbe287de735b294a1d9747d2f59c22ca54ceebb7174f5a4a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e16a0074955a8d79bbe287de735b294a1d9747d2f59c22ca54ceebb7174f5a4a/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e29b1276c9ee01b351f9edf2efe1fd2654561638304ad0b116f55d3516b832ee/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e29b1276c9ee01b351f9edf2efe1fd2654561638304ad0b116f55d3516b832ee/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f02f60e185fa11f790a793a2b8a042d7b521b3c4a117b471dfeadb8a1f1cfc54/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f02f60e185fa11f790a793a2b8a042d7b521b3c4a117b471dfeadb8a1f1cfc54/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f05d75fa7171d422ee7851daa8ca8d3543fdf8ecc1043d80c30e26d7ef0e46de/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f05d75fa7171d422ee7851daa8ca8d3543fdf8ecc1043d80c30e26d7ef0e46de/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/00a412a1-3764-404b-9e6d-3a2bbb200781/volumes/kubernetes.io~projected/kube-api-access-9pp5r:{mountpoint:/var/lib/kubelet/pods/00a412a1-3764-404b-9e6d-3a2bbb200781/volumes/kubernetes.io~projected/kube-api-access-9pp5r major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11d16990-ce0c-458f-ab1e-e9a3fed3221b/volumes/kubernetes.io~projected/kube-api-access-8n295:{mountpoint:/var/lib/kubelet/pods/11d16990-ce0c-458f-ab1e-e9a3fed3221b/volumes/kubernetes.io~projected/kube-api-access-8n295 major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1da80d21-c62b-44ec-a3fe-64633b5de401/volumes/kubernetes.io~projected/kube-api-access-nr24x:{mountpoint:/var/lib/kubelet/pods/1da80d21-c62b-44ec-a3fe-64633b5de401/volumes/kubernetes.io~projected/kube-api-access-nr24x major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd/volumes/kubernetes.io~projected/kube-api-access-vp5h5:{mountpoint:/var/lib/kubelet/pods/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd/volumes/kubernetes.io~projected/kube-api-access-vp5h5 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/volumes/kubernetes.io~projected/kube-api-access-495nb:{mountpoint:/var/lib/kubelet/pods/307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/volumes/kubernetes.io~projected/kube-api-access-495nb major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/328dde79-0c85-47a3-b3f8-38d8cc8614c0/volumes/kubernetes.io~projected/kube-api-access-dkjrd:{mountpoint:/var/lib/kubelet/pods/328dde79-0c85-47a3-b3f8-38d8cc8614c0/volumes/kubernetes.io~projected/kube-api-access-dkjrd major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32d5aaf4-3430-4e5b-aca5-f0d720ce8657/volumes/kubernetes.io~projected/kube-api-access-nvrrb:{mountpoint:/var/lib/kubelet/pods/32d5aaf4-3430-4e5b-aca5-f0d720ce8657/volumes/kubernetes.io~projected/kube-api-access-nvrrb major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32d5aaf4-3430-4e5b-aca5-f0d720ce8657/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/32d5aaf4-3430-4e5b-aca5-f0d720ce8657/volumes/kubernetes.io~secret/signing-key major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36065516-87ab-4057-b198-439305606d12/volumes/kubernetes.io~projected/kube-api-access-8k4jz:{mountpoint:/var/lib/kubelet/pods/36065516-87ab-4057-b198-439305606d12/volumes/kubernetes.io~projected/kube-api-access-8k4jz major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36065516-87ab-4057-b198-439305606d12/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/36065516-87ab-4057-b198-439305606d12/volumes/kubernetes.io~secret/default-certificate major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36065516-87ab-4057-b198-439305606d12/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/36065516-87ab-4057-b198-439305606d12/volumes/kubernetes.io~secret/stats-auth major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/396e7be7-0e72-4c32-a04a-22bd9bd708e1/volumes/kubernetes.io~projected/kube-api-access-fmwwz:{mountpoint:/var/lib/kubelet/pods/396e7be7-0e72-4c32-a04a-22bd9bd708e1/volumes/kubernetes.io~projected/kube-api-access-fmwwz major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee/volumes/kubernetes.io~projected/kube-api-access-hdggk:{mountpoint:/var/lib/kubelet/pods/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee/volumes/kubernetes.io~projected/kube-api-access-hdggk major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fe332c5-8991-4d94-9b67-c9017f4945b8/volumes/kubernetes.io~projected/kube-api-access-xl2f4:{mountpoint:/var/lib/kubelet/pods/3fe332c5-8991-4d94-9b67-c9017f4945b8/volumes/kubernetes.io~projected/kube-api-access-xl2f4 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47c859b2-a25b-45e5-acdc-0632d472c097/volumes/kubernetes.io~projected/kube-api-access-pd5tr:{mountpoint:/var/lib/kubelet/pods/47c859b2-a25b-45e5-acdc-0632d472c097/volumes/kubernetes.io~projected/kube-api-access-pd5tr major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d66aab2-4671-4bbd-a4cc-b9dd9c5a766d/volumes/kubernetes.io~projected/kube-api-access-vktv4:{mountpoint:/var/lib/kubelet/pods/4d66aab2-4671-4bbd-a4cc-b9dd9c5a766d/volumes/kubernetes.io~projected/kube-api-access-vktv4 major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60d2d1a6-a89f-4a9a-acb9-e85ce2c8f6ce/volumes/kubernetes.io~projected/kube-api-access-w5qqj:{mountpoint:/var/lib/kubelet/pods/60d2d1a6-a89f-4a9a-acb9-e85ce2c8f6ce/volumes/kubernetes.io~projected/kube-api-access-w5qqj major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61060804-c519-490a-ac90-674792d50174/volumes/kubernetes.io~projected/kube-api-access-h4w2t:{mountpoint:/var/lib/kubelet/pods/61060804-c519-490a-ac90-674792d50174/volumes/kubernetes.io~projected/kube-api-access-h4w2t major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63bfb9d0-9e87-44c6-a6f7-2f7b992e20f6/volumes/kubernetes.io~projected/kube-api-access-hrqb5:{mountpoint:/var/lib/kubelet/pods/63bfb9d0-9e87-44c6-a6f7-2f7b992e20f6/volumes/kubernetes.io~projected/kube-api-access-hrqb5 major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63bfb9d0-9e87-44c6-a6f7-2f7b992e20f6/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/63bfb9d0-9e87-44c6-a6f7-2f7b992e20f6/volumes/kubernetes.io~secret/cert major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce/volumes/kubernetes.io~projected/kube-api-access-g7782:{mountpoint:/var/lib/kubelet/pods/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce/volumes/kubernetes.io~projected/kube-api-access-g7782 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d/volumes/kubernetes.io~projected/kube-api-access-m5m26:{mountpoint:/var/lib/kubelet/pods/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d/volumes/kubernetes.io~projected/kube-api-access-m5m26 major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/840ddc67-db99-4d3f-ba2d-51632ac3912c/volumes/kubernetes.io~projected/kube-api-access-626zm:{mountpoint:/var/lib/kubelet/pods/840ddc67-db99-4d3f-ba2d-51632ac3912c/volumes/kubernetes.io~projected/kube-api-access-626zm major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d74c5e4-f631-4eae-84d5-84749b53ccf8/volumes/kubernetes.io~projected/kube-api-access-nwf5m:{mountpoint:/var/lib/kubelet/pods/8d74c5e4-f631-4eae-84d5-84749b53ccf8/volumes/kubernetes.io~projected/kube-api-access-nwf5m major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d74c5e4-f631-4eae-84d5-84749b53ccf8/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/8d74c5e4-f631-4eae-84d5-84749b53ccf8/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6466fc4-ed3d-4031-ba11-5606bd1af74b/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/a6466fc4-ed3d-4031-ba11-5606bd1af74b/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6466fc4-ed3d-4031-ba11-5606bd1af74b/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/a6466fc4-ed3d-4031-ba11-5606bd1af74b/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6466fc4-ed3d-4031-ba11-5606bd1af74b/volumes/kubernetes.io~projected/kube-api-access-s6kdm:{mountpoint:/var/lib/kubelet/pods/a6466fc4-ed3d-4031-ba11-5606bd1af74b/volumes/kubernetes.io~projected/kube-api-access-s6kdm major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b3260ef2-8ff4-4b70-a3a0-e5386a27d210/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/b3260ef2-8ff4-4b70-a3a0-e5386a27d210/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b3260ef2-8ff4-4b70-a3a0-e5386a27d210/volumes/kubernetes.io~projected/kube-api-access-79cqw:{mountpoint:/var/lib/kubelet/pods/b3260ef2-8ff4-4b70-a3a0-e5386a27d210/volumes/kubernetes.io~projected/kube-api-access-79cqw major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b3260ef2-8ff4-4b70-a3a0-e5386a27d210/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/b3260ef2-8ff4-4b70-a3a0-e5386a27d210/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b3260ef2-8ff4-4b70-a3a0-e5386a27d210/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/b3260ef2-8ff4-4b70-a3a0-e5386a27d210/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bdf183c4-a9ee-4ea8-ace1-50f307114328/volumes/kubernetes.io~projected/kube-api-access-vtr9w:{mountpoint:/var/lib/kubelet/pods/bdf183c4-a9ee-4ea8-ace1-50f307114328/volumes/kubernetes.io~projected/kube-api-access-vtr9w major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf085d57-4ae4-4add-9402-9ee1ce5d795a/volumes/kubernetes.io~projected/kube-api-access-pbw2q:{mountpoint:/var/lib/kubelet/pods/bf085d57-4ae4-4add-9402-9ee1ce5d795a/volumes/kubernetes.io~projected/kube-api-access-pbw2q major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf085d57-4ae4-4add-9402-9ee1ce5d795a/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/bf085d57-4ae4-4add-9402-9ee1ce5d795a/volumes/kubernetes.io~secret/serving-cert major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d533df4c-38b2-42b2-9cb2-b82afac33795/volumes/kubernetes.io~projected/kube-api-access-sfv25:{mountpoint:/var/lib/kubelet/pods/d533df4c-38b2-42b2-9cb2-b82afac33795/volumes/kubernetes.io~projected/kube-api-access-sfv25 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd3829b5-04dc-4112-b2d9-087d136ba9ec/volumes/kubernetes.io~projected/kube-api-access-f6m64:{mountpoint:/var/lib/kubelet/pods/dd3829b5-04dc-4112-b2d9-087d136ba9ec/volumes/kubernetes.io~projected/kube-api-access-f6m64 major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd3829b5-04dc-4112-b2d9-087d136ba9ec/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/dd3829b5-04dc-4112-b2d9-087d136ba9ec/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e637e650-5fc6-4adb-a97a-47d924c30f97/volumes/kubernetes.io~projected/kube-api-access-mr7zc:{mountpoint:/var/lib/kubelet/pods/e637e650-5fc6-4adb-a97a-47d924c30f97/volumes/kubernetes.io~projected/kube-api-access-mr7zc major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e68be0c4-d4ee-4da5-82e3-82372f811841/volumes/kubernetes.io~projected/kube-api-access-wgm54:{mountpoint:/var/lib/kubelet/pods/e68be0c4-d4ee-4da5-82e3-82372f811841/volumes/kubernetes.io~projected/kube-api-access-wgm54 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9dee2ef-6437-4963-8e27-da5fb9c75928/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/e9dee2ef-6437-4963-8e27-da5fb9c75928/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea7bdafa-8d46-4b28-b93b-b73352b2e269/volumes/kubernetes.io~projected/kube-api-access-mrn2g:{mountpoint:/var/lib/kubelet/pods/ea7bdafa-8d46-4b28-b93b-b73352b2e269/volumes/kubernetes.io~projected/kube-api-access-mrn2g major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef43449c-b7dc-4e84-b785-63201f80d4da/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/ef43449c-b7dc-4e84-b785-63201f80d4da/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f52cae01-7951-4d9e-9a57-9ba5bcdf6857/volumes/kubernetes.io~projected/kube-api-access-fkrl8:{mountpoint:/var/lib/kubelet/pods/f52cae01-7951-4d9e-9a57-9ba5bcdf6857/volumes/kubernetes.io~projected/kube-api-access-fkrl8 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fbfeab8a-b74e-4188-b565-b56442a01b79/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/fbfeab8a-b74e-4188-b565-b56442a01b79/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fbfeab8a-b74e-4188-b565-b56442a01b79/volumes/kubernetes.io~projected/kube-api-access-fbt8t:{mountpoint:/var/lib/kubelet/pods/fbfeab8a-b74e-4188-b565-b56442a01b79/volumes/kubernetes.io~projected/kube-api-access-fbt8t major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fbfeab8a-b74e-4188-b565-b56442a01b79/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/fbfeab8a-b74e-4188-b565-b56442a01b79/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fbfeab8a-b74e-4188-b565-b56442a01b79/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/fbfeab8a-b74e-4188-b565-b56442a01b79/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:205 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/0cbe45b4c39af51224d895dbe0daa90979dba0d9032498b69b398221662e64cb/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/710c6a2d74bf00b216d07a5d2e5dc50ffd36c2af49691176ca5a72ed7d812ed7/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/3aa94e27cbd9a28d7c37ef727e7a197507f5f266d67efafea0af173fb1c90957/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/7912c41613aff5b766f54fb376b9505caf7a2a18a32e90e48d3a61005460151b/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/7e42af6d93997d34e6fef9b85824f0e1125cc80caf3e5ec070b7446c713105f7/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/3a5d2c83a65806cfbbb9be94bc56e34c11ce97ba42b5643f15c900735f7501bc/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/6ba35de20780e0e96fe35680445e122ae25e4046832693ad40a3a0ceb4d075e8/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/25f52904d937382bcf3a5992a54a3132699899a8d478fc66e20d1f825ba9390f/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/b41c8010464553735130638065d4319727096a76c74567fed58a81c6f59bd3a1/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-133:{mountpoint:/var/lib/containers/storage/overlay/1ecf725cfdc20d06c8f46da0992fcab479c08b9437f8dfe4a246a4abcceeda1f/merged major:0 minor:133 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/6b8623d4ee6d4c94a1d2e59e0e178467333ddfe2f19c8d18645ccc89673500aa/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/b4ef0049e3daa9536f4e4138eb632b500e747ab24b4c6451a95b1938ce5ba731/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/29f93f945b4fab0130a1f6380e5290a50d2ecd1fd178fd5b5b31e872c85ac0ec/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/68fb29f383acc7b135fe5cdbf7b20017ec5ba47adead2fb25a8fca97cc4eb3d8/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/d6ab0b622f0da891c8af860bf7f614813553ae1da925dc4376b3fac363dec52b/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/af5902b2b5b9e434c22dbc264ce2a56d7eef08649151a6425e31e7d57dcbd437/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/1f993192088f1d127cb0e4a73b8599779f83c90761a00e427686527a095327dc/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/1b5d132c0a6337dbb6df9712451d26aa30ada2b54eaddd94439f78c8f34e9f6d/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/d3deb53fbc5954b73af75425bca89d42783c8b634950a69edb39b405d287e781/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/5ba19d791b64d9e25d7a80baee39c66af12d0268bd7d108ac49289f826f9ad67/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/8faaf65339386a70e35ddf5edc756f8eef5f7f8efd9157b1c68f14752bdf0162/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/ea8e3dd8e38476ef255808e4832948af59335ecfa9669aba7e952b96afc1a2ed/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/b3a5700292e7ad9cd079e11383c99a808442afa991f3d0874fae9b8ed3bebac3/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/d83d635a1d1205e2a4fb242ff17eb105de1f21bca6217e95301702262b0de8d5/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/e12117bc883df372674b1b5d05ec3074948cd8c372c2a83dde54460392967287/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/3263ec32a909dfcc45476364488eaa89a4536533d42f8af00f4ade81665f4a6c/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/271574b5641f4158d08bc50d71133bab11c288474a81919f28a3ef3f9fd1bcdc/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/74d929e35d8a332ca7bbf4fdd8aff326171da9ef66d5657e6f70d9f08a4d4cbe/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/8563d6ba70a5cab2f9a4fe9c210e6df798a15f74a21cc0183af6165c0ce28d9c/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/f88d9c5b5ed4f2957974f7f2e0d3313647170a5beadcbb6d0d1964d3888db519/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/a2b6298c9bc54e679b23dc987192844ce8a79a3c8f48ca94c1921d62769ac717/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/340960c4b2ff2989f407f6625184f79aa1d94af14a3a60ee3725d5afe0389470/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/d256e7a0c82fca6242b9beedc4400a1aa03b3526fb26de0816e6a0fbd1936f88/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/0dc4c6f081584da41242d62b5d18b467bef73fabc220212a5048c73c772c2c07/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/ce5685d0be41305f3afc0f31045d6c928b9fcbed0e3e5131444438f5bf2c27d7/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-297:{mountpoint:/var/lib/containers/storage/overlay/27b11f0308572c9bb9c454286155dac5f7fea67480fc3a731c6f3a1bdb679de0/merged major:0 minor:297 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/783085be42b5a929a50112203e73130a0cfc5906dd346ec87a243ad2694bb0a7/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/785fba370942f9fbbb897ec09cdb3d5c83c6ab3679f1a2023d347758116615f4/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-303:{mountpoint:/var/lib/containers/storage/overlay/a0a9adc803eec48449847049c18c2152083a61efa7a1d7b59edafa0fa52f8c2f/merged major:0 minor:303 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/df53e54c06844a9dd5647456db184b0d53a9f51791abf2419ddc5badec6456c2/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/7e53fb95b4e4b187403754d096fb5b7159b73829ac33ed791d9fb51c79d6a8d2/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/972d754a7cd518f3d9022a7bad0858bcf9407b5721e9e4397c750e165fa35b6d/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/4f2a369233b4a7a402d675f9295d223844226311e87c960890786f4fed12dce1/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/178f031d86dfe77188ae38364a90c7c15989b08016d61798a5d698c90893bfad/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/2ff40ace64b9b9342887b52b8958bbcd2949f26a08a47bafed5f5ad9a737f0d5/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/d7ed524dbafc3023de99d9edf544229183a09bc736809cb52a3566ad36f719e5/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/d7a406a49751599eabbc0741f10a35a4eea26cd4374019111de1ba07bea4f5a2/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/c3cbb26c2fd7bbec9afb2a1fbd9820924b0c194a247761141f6d2eb0c30a98b3/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/4a0e2f94b2e0b9042258950fa05e6d3cf90b6eacaf37278039614ffce2c647c3/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/8fdadb6c7cfd2be0d031bc2aaf3dae88b172a9a2cdd044c0fe48bfbe6c1e0254/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/21b981aac1cd701b114c2c8db65236afbb4292d546e2601c80a2981c61d83a91/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-372:{mountpoint:/var/lib/containers/storage/overlay/8d719fb9fe2848ba33253dd286cbeb02384595eecee885e75a767075e58af476/merged major:0 minor:372 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/9c5b7344a72b640d65224a8f932ab23c7c454a7a7240ebe1817e406fc6f36b7e/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/9d577f185d1cb275dcb0584e15439e574bfb1cb4d5ce8d0478570735c8fd8027/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/174a26182bdcbe74a5fd9b82b14ec5cc39913e8c2792bad0e30d5416a6ae9c4b/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/bcd91eba183268c15e7ee7b4899de8d1f2b382edc35174bce8ab687db578a41a/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/359ca6c80d0f6884f652531767f33db6714cf1d9056301993e130b586956c4e2/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/662aff4c4f9cb41c12d0d0a16f4588aff95b85739f109fccda4c82ff6d56ccee/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/f24b134cd16b026c0fe09a4aa2f390fe16b20d3d453ff5a3b8718a21b1362b37/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/8634eb80478e8a960c23c21dc0a2aea4e83bd01735c6604bd8ac80fc9538449f/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/074ae0f9187bd5b6a9321aa95a59341a47d60c067d7588dd6493e2a642d9fb8e/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/973fff5eac94203a01ab60fcf05c1c46bd3f8206f777755df278b5e5d11544d1/merged major:0 minor:98 fsType:overlay blockSize:0}] Nov 03 14:11:06.174562 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.174185 5060 manager.go:217] Machine: {Timestamp:2026-11-03 14:11:06.173694626 +0000 UTC m=+0.062484197 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098827 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec263f13a03ab563fedaf683c8594dba SystemUUID:ec263f13-a03a-b563-feda-f683c8594dba BootID:6c77de02-66f7-4bdb-ab9f-9e527abc5356 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a6466fc4-ed3d-4031-ba11-5606bd1af74b/volumes/kubernetes.io~projected/kube-api-access-s6kdm DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6267a5ca51fba195deca8d7782843172823bedf31516805702604683e8d17646/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/63bfb9d0-9e87-44c6-a6f7-2f7b992e20f6/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/14d85150d7edbe1991a5ff99576317c8d72a74e40cd748fddc80df8fbf48a7c2/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fbfeab8a-b74e-4188-b565-b56442a01b79/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bdf183c4-a9ee-4ea8-ace1-50f307114328/volumes/kubernetes.io~projected/kube-api-access-vtr9w DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b3260ef2-8ff4-4b70-a3a0-e5386a27d210/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/840ddc67-db99-4d3f-ba2d-51632ac3912c/volumes/kubernetes.io~projected/kube-api-access-626zm DeviceMajor:0 DeviceMinor:227 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-297 DeviceMajor:0 DeviceMinor:297 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2867061ef7d1b1c91f970babfd5e957a2dc2ae00bff59a673a22086f0cf86e89/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dd3829b5-04dc-4112-b2d9-087d136ba9ec/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/60d2d1a6-a89f-4a9a-acb9-e85ce2c8f6ce/volumes/kubernetes.io~projected/kube-api-access-w5qqj DeviceMajor:0 DeviceMinor:343 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/36065516-87ab-4057-b198-439305606d12/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd/volumes/kubernetes.io~projected/kube-api-access-vp5h5 DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a7a404869ab852ee9956a1299567c7d73017b8aa057e1718974ce544148abd5f/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dd3829b5-04dc-4112-b2d9-087d136ba9ec/volumes/kubernetes.io~projected/kube-api-access-f6m64 DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bf085d57-4ae4-4add-9402-9ee1ce5d795a/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f05d75fa7171d422ee7851daa8ca8d3543fdf8ecc1043d80c30e26d7ef0e46de/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f02f60e185fa11f790a793a2b8a042d7b521b3c4a117b471dfeadb8a1f1cfc54/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/63bfb9d0-9e87-44c6-a6f7-2f7b992e20f6/volumes/kubernetes.io~projected/kube-api-access-hrqb5 DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/11d16990-ce0c-458f-ab1e-e9a3fed3221b/volumes/kubernetes.io~projected/kube-api-access-8n295 DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/db44b327584d75050b083495c3de9df2448c9f260f6100e73077383104187e6b/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d/volumes/kubernetes.io~projected/kube-api-access-m5m26 DeviceMajor:0 DeviceMinor:340 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c4519af9ec99a107edf3cad6b845dcf115f26190a900862df518f5307e681b6a/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f52cae01-7951-4d9e-9a57-9ba5bcdf6857/volumes/kubernetes.io~projected/kube-api-access-fkrl8 DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e68be0c4-d4ee-4da5-82e3-82372f811841/volumes/kubernetes.io~projected/kube-api-access-wgm54 DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/36065516-87ab-4057-b198-439305606d12/volumes/kubernetes.io~projected/kube-api-access-8k4jz DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-372 DeviceMajor:0 DeviceMinor:372 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dd6604a80c529a9ea0dfffe1743aecf939cd6a48243ad6ffa3691bb8cfe13087/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d533df4c-38b2-42b2-9cb2-b82afac33795/volumes/kubernetes.io~projected/kube-api-access-sfv25 DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8d74c5e4-f631-4eae-84d5-84749b53ccf8/volumes/kubernetes.io~projected/kube-api-access-nwf5m DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ae33d6534d669fa34314d880c5da457d96e430d11244450ad159fd464d82ea95/userdata/shm DeviceMajor:0 DeviceMinor:295 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b3260ef2-8ff4-4b70-a3a0-e5386a27d210/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/32d5aaf4-3430-4e5b-aca5-f0d720ce8657/volumes/kubernetes.io~projected/kube-api-access-nvrrb DeviceMajor:0 DeviceMinor:361 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e637e650-5fc6-4adb-a97a-47d924c30f97/volumes/kubernetes.io~projected/kube-api-access-mr7zc DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8d03b1024d60e73a8391f33068de1832e67c70d38283aa74f1366514ff8452c5/userdata/shm DeviceMajor:0 DeviceMinor:335 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bf085d57-4ae4-4add-9402-9ee1ce5d795a/volumes/kubernetes.io~projected/kube-api-access-pbw2q DeviceMajor:0 DeviceMinor:228 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b6867239ea6208777242d582c882a81cd26834e80a64ddff473a7fdc1e89e0fd/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/396e7be7-0e72-4c32-a04a-22bd9bd708e1/volumes/kubernetes.io~projected/kube-api-access-fmwwz DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-133 DeviceMajor:0 DeviceMinor:133 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1da80d21-c62b-44ec-a3fe-64633b5de401/volumes/kubernetes.io~projected/kube-api-access-nr24x DeviceMajor:0 DeviceMinor:342 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/36065516-87ab-4057-b198-439305606d12/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dcb4a147277102c0ee92ed924dc29e6304010e9c9c62749f58c0cc45c6f40640/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b3260ef2-8ff4-4b70-a3a0-e5386a27d210/volumes/kubernetes.io~projected/kube-api-access-79cqw DeviceMajor:0 DeviceMinor:231 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/61060804-c519-490a-ac90-674792d50174/volumes/kubernetes.io~projected/kube-api-access-h4w2t DeviceMajor:0 DeviceMinor:230 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee/volumes/kubernetes.io~projected/kube-api-access-hdggk DeviceMajor:0 DeviceMinor:318 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d5ed3048d51d2d7d87085ac9f066e7d8e6b663be1405719040ec7c7ab734958a/userdata/shm DeviceMajor:0 DeviceMinor:346 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3f3072125365fd78ca387225052c6f2bdeaf31194b0d58f1497687c2422dee99/userdata/shm DeviceMajor:0 DeviceMinor:362 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3fe332c5-8991-4d94-9b67-c9017f4945b8/volumes/kubernetes.io~projected/kube-api-access-xl2f4 DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/00a412a1-3764-404b-9e6d-3a2bbb200781/volumes/kubernetes.io~projected/kube-api-access-9pp5r DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/13785db1c6914236913af97f8c8f3777a7a9ebf03e55a39f25ff3e08eb9c0328/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9876113882c1611251a2fc38d5a386ff1f49e928e66cd1a51b21a3d5a08542ae/userdata/shm DeviceMajor:0 DeviceMinor:293 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ea7bdafa-8d46-4b28-b93b-b73352b2e269/volumes/kubernetes.io~projected/kube-api-access-mrn2g DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce/volumes/kubernetes.io~projected/kube-api-access-g7782 DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e29b1276c9ee01b351f9edf2efe1fd2654561638304ad0b116f55d3516b832ee/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fbfeab8a-b74e-4188-b565-b56442a01b79/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/volumes/kubernetes.io~projected/kube-api-access-495nb DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ccae5bf3c9ab1e801ea85c8e8dea88e7011d7b98b5015e975f65b768ea2eb6c3/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9c4f3c61cca7f03a33d5ad5e9a4a82825d334b9c05f2baebe6ab49f2dee6d18b/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2ef0fa9513f22e1f5925f997bf3f1f4eeb9577ab139104228f84d2dd73133f20/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fbfeab8a-b74e-4188-b565-b56442a01b79/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-303 DeviceMajor:0 DeviceMinor:303 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ef43449c-b7dc-4e84-b785-63201f80d4da/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e9dee2ef-6437-4963-8e27-da5fb9c75928/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8d74c5e4-f631-4eae-84d5-84749b53ccf8/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b3260ef2-8ff4-4b70-a3a0-e5386a27d210/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1da2eca1baf31392b9832da78c58b8a61a6e04b0b418d6920124b73c4c7d4b03/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e16a0074955a8d79bbe287de735b294a1d9747d2f59c22ca54ceebb7174f5a4a/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/32d5aaf4-3430-4e5b-aca5-f0d720ce8657/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:360 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/328dde79-0c85-47a3-b3f8-38d8cc8614c0/volumes/kubernetes.io~projected/kube-api-access-dkjrd DeviceMajor:0 DeviceMinor:300 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ddafe78a5f8133c3e0127472345a985b2211323b1eefc6f97145d0d119c24cd2/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a6466fc4-ed3d-4031-ba11-5606bd1af74b/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a6466fc4-ed3d-4031-ba11-5606bd1af74b/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4d66aab2-4671-4bbd-a4cc-b9dd9c5a766d/volumes/kubernetes.io~projected/kube-api-access-vktv4 DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/47c859b2-a25b-45e5-acdc-0632d472c097/volumes/kubernetes.io~projected/kube-api-access-pd5tr DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7640e0f73cde1a06114a434eeb5ceb533b361171bd334ae3fff87ccbb55c03ff/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fbfeab8a-b74e-4188-b565-b56442a01b79/volumes/kubernetes.io~projected/kube-api-access-fbt8t DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/430a1e4878dba2ce0ac77cbc20ec0e6f83bd85eacc21aa2aeb477f543173caca/userdata/shm DeviceMajor:0 DeviceMinor:344 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:13785db1c691423 MacAddress:6a:68:16:87:ae:75 Speed:10000 Mtu:8901} {Name:1da2eca1baf3139 MacAddress:f6:bb:1c:40:0d:f9 Speed:10000 Mtu:8901} {Name:3f3072125365fd7 MacAddress:62:6b:54:88:72:ea Speed:10000 Mtu:8901} {Name:430a1e4878dba2c MacAddress:ca:c3:7a:86:a1:1a Speed:10000 Mtu:8901} {Name:7640e0f73cde1a0 MacAddress:d2:0c:e3:29:84:5e Speed:10000 Mtu:8901} {Name:8d03b1024d60e73 MacAddress:6e:63:2b:b2:bb:c7 Speed:10000 Mtu:8901} {Name:9876113882c1611 MacAddress:82:51:a6:66:4d:24 Speed:10000 Mtu:8901} {Name:9c4f3c61cca7f03 MacAddress:36:f6:a5:6c:0b:53 Speed:10000 Mtu:8901} {Name:ae33d6534d669fa MacAddress:ba:94:82:a3:66:6f Speed:10000 Mtu:8901} {Name:b6867239ea62087 MacAddress:1e:e4:1c:d3:3b:8c Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d8:63:14:e5 Speed:0 Mtu:9001} {Name:br-int MacAddress:26:77:e8:7a:01:09 Speed:0 Mtu:8901} {Name:ccae5bf3c9ab1e8 MacAddress:9e:da:c7:86:bd:e9 Speed:10000 Mtu:8901} {Name:d5ed3048d51d2d7 MacAddress:de:4d:d1:e8:89:54 Speed:10000 Mtu:8901} {Name:db44b327584d750 MacAddress:12:3c:2d:14:50:45 Speed:10000 Mtu:8901} {Name:dcb4a147277102c MacAddress:4a:87:f1:a3:01:b2 Speed:10000 Mtu:8901} {Name:e16a0074955a8d7 MacAddress:2e:d6:4a:cd:ed:db Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d8:63:14:e5 Speed:0 Mtu:9001} {Name:f02f60e185fa11f MacAddress:6e:89:48:1a:cf:d6 Speed:10000 Mtu:8901} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:86:93:f7:8e:47:3b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 14:11:06.174770 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.174565 5060 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 14:11:06.174770 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.174609 5060 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 14:11:06.174918 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.174898 5060 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 14:11:06.175037 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.174918 5060 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-171.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 14:11:06.175081 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.175039 5060 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 14:11:06.175081 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.175047 5060 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 14:11:06.175081 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.175057 5060 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 14:11:06.175081 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.175076 5060 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 14:11:06.175184 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.175096 5060 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 14:11:06.175241 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.175231 5060 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 14:11:06.175359 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.175350 5060 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 14:11:06.175390 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.175386 5060 kubelet.go:493] "Attempting to sync node with API server" Nov 03 14:11:06.175418 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.175395 5060 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 14:11:06.175418 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.175405 5060 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 14:11:06.175418 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.175414 5060 kubelet.go:405] "Adding apiserver pod source" Nov 03 14:11:06.175506 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.175421 5060 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 14:11:06.176310 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.176296 5060 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 14:11:06.176310 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.176309 5060 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 14:11:06.176863 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.176830 5060 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 14:11:06.179459 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.179447 5060 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 14:11:06.179459 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.179459 5060 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 14:11:06.179569 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.179560 5060 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 14:11:06.179757 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.179746 5060 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 14:11:06.179788 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.179776 5060 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 14:11:06.179906 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.179897 5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 14:11:06.179938 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.179911 5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 14:11:06.179938 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.179917 5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 14:11:06.179938 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.179923 5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 14:11:06.179938 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.179929 5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 14:11:06.180047 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.179939 5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 14:11:06.180047 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.179948 5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 14:11:06.180047 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.179954 5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 14:11:06.180047 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.179961 5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 14:11:06.180047 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.179966 5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 14:11:06.180047 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.179974 5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 14:11:06.180047 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.179983 5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 14:11:06.180047 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.180005 5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 14:11:06.180047 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.180015 5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 14:11:06.183560 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.183543 5060 server.go:1263] "Started kubelet" Nov 03 14:11:06.184092 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.183762 5060 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 14:11:06.184092 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.183660 5060 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 14:11:06.184092 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.183960 5060 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 14:11:06.193705 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.184326 5060 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 14:11:06.193705 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.185411 5060 server.go:317] "Adding debug handlers to kubelet server" Nov 03 14:11:06.193705 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.190087 5060 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 14:11:06.193705 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.190222 5060 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 14:11:06.193705 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.190487 5060 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 14:05:26 +0000 UTC" deadline="2028-06-17 17:46:06.118188315 +0000 UTC" Nov 03 14:11:06.193705 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.190523 5060 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 14:11:06.193705 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.190533 5060 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 14:11:06.193705 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.190556 5060 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 14:11:06.193705 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.190522 5060 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14211h34m59.927670556s" Nov 03 14:11:06.193705 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:06.192788 5060 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-171.ec2.internal\" not found" Nov 03 14:11:06.184175 ip-10-0-143-171 systemd[1]: Started Kubernetes Kubelet. Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196180 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea7bdafa-8d46-4b28-b93b-b73352b2e269" volumeName="kubernetes.io/configmap/ea7bdafa-8d46-4b28-b93b-b73352b2e269-samples-operator-config" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196229 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d" volumeName="kubernetes.io/configmap/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d-service-ca-bundle" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196245 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d" volumeName="kubernetes.io/empty-dir/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d-archives-path" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196258 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9dee2ef-6437-4963-8e27-da5fb9c75928" volumeName="kubernetes.io/configmap/e9dee2ef-6437-4963-8e27-da5fb9c75928-konnectivity-ca" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196271 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbfeab8a-b74e-4188-b565-b56442a01b79" volumeName="kubernetes.io/empty-dir/fbfeab8a-b74e-4188-b565-b56442a01b79-ca-trust-extracted" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196287 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbfeab8a-b74e-4188-b565-b56442a01b79" volumeName="kubernetes.io/secret/fbfeab8a-b74e-4188-b565-b56442a01b79-image-registry-private-configuration" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196301 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="307e48f6-4a66-4c9c-a4ff-848cce9d3e0e" volumeName="kubernetes.io/configmap/307e48f6-4a66-4c9c-a4ff-848cce9d3e0e-config" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196313 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="396e7be7-0e72-4c32-a04a-22bd9bd708e1" volumeName="kubernetes.io/projected/396e7be7-0e72-4c32-a04a-22bd9bd708e1-kube-api-access-fmwwz" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196342 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee" volumeName="kubernetes.io/projected/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-kube-api-access-hdggk" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196355 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fe332c5-8991-4d94-9b67-c9017f4945b8" volumeName="kubernetes.io/configmap/3fe332c5-8991-4d94-9b67-c9017f4945b8-cni-binary-copy" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196369 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fe332c5-8991-4d94-9b67-c9017f4945b8" volumeName="kubernetes.io/projected/3fe332c5-8991-4d94-9b67-c9017f4945b8-kube-api-access-xl2f4" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196382 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61060804-c519-490a-ac90-674792d50174" volumeName="kubernetes.io/projected/61060804-c519-490a-ac90-674792d50174-kube-api-access-h4w2t" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196395 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d74c5e4-f631-4eae-84d5-84749b53ccf8" volumeName="kubernetes.io/configmap/8d74c5e4-f631-4eae-84d5-84749b53ccf8-config" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196408 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d74c5e4-f631-4eae-84d5-84749b53ccf8" volumeName="kubernetes.io/projected/8d74c5e4-f631-4eae-84d5-84749b53ccf8-kube-api-access-nwf5m" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196424 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11d16990-ce0c-458f-ab1e-e9a3fed3221b" volumeName="kubernetes.io/projected/11d16990-ce0c-458f-ab1e-e9a3fed3221b-kube-api-access-8n295" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196443 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd" volumeName="kubernetes.io/configmap/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196456 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32d5aaf4-3430-4e5b-aca5-f0d720ce8657" volumeName="kubernetes.io/configmap/32d5aaf4-3430-4e5b-aca5-f0d720ce8657-signing-cabundle" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196469 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36065516-87ab-4057-b198-439305606d12" volumeName="kubernetes.io/secret/36065516-87ab-4057-b198-439305606d12-stats-auth" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196482 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d66aab2-4671-4bbd-a4cc-b9dd9c5a766d" volumeName="kubernetes.io/projected/4d66aab2-4671-4bbd-a4cc-b9dd9c5a766d-kube-api-access-vktv4" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196495 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce" volumeName="kubernetes.io/configmap/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-ovnkube-config" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196507 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce" volumeName="kubernetes.io/projected/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-kube-api-access-g7782" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196518 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6466fc4-ed3d-4031-ba11-5606bd1af74b" volumeName="kubernetes.io/empty-dir/a6466fc4-ed3d-4031-ba11-5606bd1af74b-etc-tuned" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196532 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60d2d1a6-a89f-4a9a-acb9-e85ce2c8f6ce" volumeName="kubernetes.io/projected/60d2d1a6-a89f-4a9a-acb9-e85ce2c8f6ce-kube-api-access-w5qqj" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196546 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd" volumeName="kubernetes.io/configmap/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196563 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="307e48f6-4a66-4c9c-a4ff-848cce9d3e0e" volumeName="kubernetes.io/secret/307e48f6-4a66-4c9c-a4ff-848cce9d3e0e-serving-cert" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196575 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61060804-c519-490a-ac90-674792d50174" volumeName="kubernetes.io/configmap/61060804-c519-490a-ac90-674792d50174-config-volume" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196587 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3260ef2-8ff4-4b70-a3a0-e5386a27d210" volumeName="kubernetes.io/empty-dir/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-ca-trust-extracted" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196610 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdf183c4-a9ee-4ea8-ace1-50f307114328" volumeName="kubernetes.io/projected/bdf183c4-a9ee-4ea8-ace1-50f307114328-kube-api-access-vtr9w" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196630 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e637e650-5fc6-4adb-a97a-47d924c30f97" volumeName="kubernetes.io/empty-dir/e637e650-5fc6-4adb-a97a-47d924c30f97-tmp-dir" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196643 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef43449c-b7dc-4e84-b785-63201f80d4da" volumeName="kubernetes.io/secret/ef43449c-b7dc-4e84-b785-63201f80d4da-original-pull-secret" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196654 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6466fc4-ed3d-4031-ba11-5606bd1af74b" volumeName="kubernetes.io/empty-dir/a6466fc4-ed3d-4031-ba11-5606bd1af74b-tmp" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196667 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6466fc4-ed3d-4031-ba11-5606bd1af74b" volumeName="kubernetes.io/projected/a6466fc4-ed3d-4031-ba11-5606bd1af74b-kube-api-access-s6kdm" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196681 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d533df4c-38b2-42b2-9cb2-b82afac33795" volumeName="kubernetes.io/projected/d533df4c-38b2-42b2-9cb2-b82afac33795-kube-api-access-sfv25" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196694 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd3829b5-04dc-4112-b2d9-087d136ba9ec" volumeName="kubernetes.io/empty-dir/dd3829b5-04dc-4112-b2d9-087d136ba9ec-tmp" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196707 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e68be0c4-d4ee-4da5-82e3-82372f811841" volumeName="kubernetes.io/configmap/e68be0c4-d4ee-4da5-82e3-82372f811841-serviceca" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196719 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea7bdafa-8d46-4b28-b93b-b73352b2e269" volumeName="kubernetes.io/projected/ea7bdafa-8d46-4b28-b93b-b73352b2e269-kube-api-access-mrn2g" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196732 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f52cae01-7951-4d9e-9a57-9ba5bcdf6857" volumeName="kubernetes.io/projected/f52cae01-7951-4d9e-9a57-9ba5bcdf6857-kube-api-access-fkrl8" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196745 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbfeab8a-b74e-4188-b565-b56442a01b79" volumeName="kubernetes.io/configmap/fbfeab8a-b74e-4188-b565-b56442a01b79-registry-certificates" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196763 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd" volumeName="kubernetes.io/configmap/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd-cni-binary-copy" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196775 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36065516-87ab-4057-b198-439305606d12" volumeName="kubernetes.io/secret/36065516-87ab-4057-b198-439305606d12-default-certificate" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196788 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63bfb9d0-9e87-44c6-a6f7-2f7b992e20f6" volumeName="kubernetes.io/secret/63bfb9d0-9e87-44c6-a6f7-2f7b992e20f6-cert" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196800 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d74c5e4-f631-4eae-84d5-84749b53ccf8" volumeName="kubernetes.io/secret/8d74c5e4-f631-4eae-84d5-84749b53ccf8-serving-cert" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196820 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3260ef2-8ff4-4b70-a3a0-e5386a27d210" volumeName="kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-kube-api-access-79cqw" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196848 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3260ef2-8ff4-4b70-a3a0-e5386a27d210" volumeName="kubernetes.io/secret/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-installation-pull-secrets" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196863 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e68be0c4-d4ee-4da5-82e3-82372f811841" volumeName="kubernetes.io/projected/e68be0c4-d4ee-4da5-82e3-82372f811841-kube-api-access-wgm54" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196874 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbfeab8a-b74e-4188-b565-b56442a01b79" volumeName="kubernetes.io/secret/fbfeab8a-b74e-4188-b565-b56442a01b79-installation-pull-secrets" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196886 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61060804-c519-490a-ac90-674792d50174" volumeName="kubernetes.io/empty-dir/61060804-c519-490a-ac90-674792d50174-tmp-dir" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196897 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d" volumeName="kubernetes.io/configmap/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d-trusted-ca-bundle" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196908 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd3829b5-04dc-4112-b2d9-087d136ba9ec" volumeName="kubernetes.io/projected/dd3829b5-04dc-4112-b2d9-087d136ba9ec-kube-api-access-f6m64" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196922 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e637e650-5fc6-4adb-a97a-47d924c30f97" volumeName="kubernetes.io/projected/e637e650-5fc6-4adb-a97a-47d924c30f97-kube-api-access-mr7zc" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196933 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3260ef2-8ff4-4b70-a3a0-e5386a27d210" volumeName="kubernetes.io/configmap/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-trusted-ca" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196944 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="396e7be7-0e72-4c32-a04a-22bd9bd708e1" volumeName="kubernetes.io/configmap/396e7be7-0e72-4c32-a04a-22bd9bd708e1-iptables-alerter-script" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196957 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd" volumeName="kubernetes.io/projected/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd-kube-api-access-vp5h5" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196970 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36065516-87ab-4057-b198-439305606d12" volumeName="kubernetes.io/projected/36065516-87ab-4057-b198-439305606d12-kube-api-access-8k4jz" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196984 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f219713-6fd8-4580-b1c0-24e2a929c258" volumeName="kubernetes.io/configmap/6f219713-6fd8-4580-b1c0-24e2a929c258-nginx-conf" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196998 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3260ef2-8ff4-4b70-a3a0-e5386a27d210" volumeName="kubernetes.io/secret/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-image-registry-private-configuration" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.197015 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd3829b5-04dc-4112-b2d9-087d136ba9ec" volumeName="kubernetes.io/configmap/dd3829b5-04dc-4112-b2d9-087d136ba9ec-trusted-ca-bundle" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.197028 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd3829b5-04dc-4112-b2d9-087d136ba9ec" volumeName="kubernetes.io/secret/dd3829b5-04dc-4112-b2d9-087d136ba9ec-serving-cert" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.197039 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fe332c5-8991-4d94-9b67-c9017f4945b8" volumeName="kubernetes.io/configmap/3fe332c5-8991-4d94-9b67-c9017f4945b8-multus-daemon-config" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.197047 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce" volumeName="kubernetes.io/configmap/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-env-overrides" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.197054 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d" volumeName="kubernetes.io/projected/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d-kube-api-access-m5m26" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.197065 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32d5aaf4-3430-4e5b-aca5-f0d720ce8657" volumeName="kubernetes.io/secret/32d5aaf4-3430-4e5b-aca5-f0d720ce8657-signing-key" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.197075 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47c859b2-a25b-45e5-acdc-0632d472c097" volumeName="kubernetes.io/projected/47c859b2-a25b-45e5-acdc-0632d472c097-kube-api-access-pd5tr" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.197085 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce" volumeName="kubernetes.io/secret/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.197097 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdf183c4-a9ee-4ea8-ace1-50f307114328" volumeName="kubernetes.io/configmap/bdf183c4-a9ee-4ea8-ace1-50f307114328-telemetry-config" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.197104 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00a412a1-3764-404b-9e6d-3a2bbb200781" volumeName="kubernetes.io/projected/00a412a1-3764-404b-9e6d-3a2bbb200781-kube-api-access-9pp5r" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.197111 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbfeab8a-b74e-4188-b565-b56442a01b79" volumeName="kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-bound-sa-token" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.197123 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1da80d21-c62b-44ec-a3fe-64633b5de401" volumeName="kubernetes.io/projected/1da80d21-c62b-44ec-a3fe-64633b5de401-kube-api-access-nr24x" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.197133 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="307e48f6-4a66-4c9c-a4ff-848cce9d3e0e" volumeName="kubernetes.io/projected/307e48f6-4a66-4c9c-a4ff-848cce9d3e0e-kube-api-access-495nb" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.197141 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="307e48f6-4a66-4c9c-a4ff-848cce9d3e0e" volumeName="kubernetes.io/configmap/307e48f6-4a66-4c9c-a4ff-848cce9d3e0e-trusted-ca" seLinuxMountContext="" Nov 03 14:11:06.197084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.197148 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf085d57-4ae4-4add-9402-9ee1ce5d795a" volumeName="kubernetes.io/secret/bf085d57-4ae4-4add-9402-9ee1ce5d795a-serving-cert" seLinuxMountContext="" Nov 03 14:11:06.200256 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.197156 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd3829b5-04dc-4112-b2d9-087d136ba9ec" volumeName="kubernetes.io/empty-dir/dd3829b5-04dc-4112-b2d9-087d136ba9ec-snapshots" seLinuxMountContext="" Nov 03 14:11:06.200256 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.197163 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbfeab8a-b74e-4188-b565-b56442a01b79" volumeName="kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-kube-api-access-fbt8t" seLinuxMountContext="" Nov 03 14:11:06.200256 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.197170 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63bfb9d0-9e87-44c6-a6f7-2f7b992e20f6" volumeName="kubernetes.io/projected/63bfb9d0-9e87-44c6-a6f7-2f7b992e20f6-kube-api-access-hrqb5" seLinuxMountContext="" Nov 03 14:11:06.200256 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.197179 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf085d57-4ae4-4add-9402-9ee1ce5d795a" volumeName="kubernetes.io/configmap/bf085d57-4ae4-4add-9402-9ee1ce5d795a-config" seLinuxMountContext="" Nov 03 14:11:06.200256 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.197190 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd3829b5-04dc-4112-b2d9-087d136ba9ec" volumeName="kubernetes.io/configmap/dd3829b5-04dc-4112-b2d9-087d136ba9ec-service-ca-bundle" seLinuxMountContext="" Nov 03 14:11:06.200256 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.197197 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9dee2ef-6437-4963-8e27-da5fb9c75928" volumeName="kubernetes.io/secret/e9dee2ef-6437-4963-8e27-da5fb9c75928-agent-certs" seLinuxMountContext="" Nov 03 14:11:06.200256 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.197205 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3260ef2-8ff4-4b70-a3a0-e5386a27d210" volumeName="kubernetes.io/configmap/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-registry-certificates" seLinuxMountContext="" Nov 03 14:11:06.200256 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.197212 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf085d57-4ae4-4add-9402-9ee1ce5d795a" volumeName="kubernetes.io/projected/bf085d57-4ae4-4add-9402-9ee1ce5d795a-kube-api-access-pbw2q" seLinuxMountContext="" Nov 03 14:11:06.200256 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.197220 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbfeab8a-b74e-4188-b565-b56442a01b79" volumeName="kubernetes.io/configmap/fbfeab8a-b74e-4188-b565-b56442a01b79-trusted-ca" seLinuxMountContext="" Nov 03 14:11:06.200256 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.197236 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="840ddc67-db99-4d3f-ba2d-51632ac3912c" volumeName="kubernetes.io/projected/840ddc67-db99-4d3f-ba2d-51632ac3912c-kube-api-access-626zm" seLinuxMountContext="" Nov 03 14:11:06.200256 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.197243 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3260ef2-8ff4-4b70-a3a0-e5386a27d210" volumeName="kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-bound-sa-token" seLinuxMountContext="" Nov 03 14:11:06.200256 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.197250 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee" volumeName="kubernetes.io/empty-dir/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-data-volume" seLinuxMountContext="" Nov 03 14:11:06.200256 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.197257 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32d5aaf4-3430-4e5b-aca5-f0d720ce8657" volumeName="kubernetes.io/projected/32d5aaf4-3430-4e5b-aca5-f0d720ce8657-kube-api-access-nvrrb" seLinuxMountContext="" Nov 03 14:11:06.200256 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.197263 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="328dde79-0c85-47a3-b3f8-38d8cc8614c0" volumeName="kubernetes.io/projected/328dde79-0c85-47a3-b3f8-38d8cc8614c0-kube-api-access-dkjrd" seLinuxMountContext="" Nov 03 14:11:06.200256 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.197270 5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce" volumeName="kubernetes.io/configmap/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-ovnkube-script-lib" seLinuxMountContext="" Nov 03 14:11:06.200256 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.197277 5060 reconstruct.go:97] "Volume reconstruction finished" Nov 03 14:11:06.200256 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.197283 5060 reconciler.go:29] "Reconciler: start to sync state" Nov 03 14:11:06.200256 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.196985 5060 factory.go:153] Registering CRI-O factory Nov 03 14:11:06.200256 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.197443 5060 factory.go:223] Registration of the crio container factory successfully Nov 03 14:11:06.200256 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.197494 5060 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 14:11:06.200256 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.197501 5060 factory.go:55] Registering systemd factory Nov 03 14:11:06.200256 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.197507 5060 factory.go:223] Registration of the systemd container factory successfully Nov 03 14:11:06.200256 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.197524 5060 factory.go:103] Registering Raw factory Nov 03 14:11:06.200256 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.197536 5060 manager.go:1199] Started watching for new ooms in manager Nov 03 14:11:06.200256 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.197776 5060 manager.go:319] Starting recovery of all containers Nov 03 14:11:06.226216 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.226182 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/1.log" Nov 03 14:11:06.227023 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.227002 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/1.log" Nov 03 14:11:06.231275 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.231225 5060 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 14:11:06.241276 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.241250 5060 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 14:11:06.241276 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.241270 5060 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 14:11:06.241276 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.241269 5060 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 14:11:06.241474 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.241287 5060 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 14:11:06.241474 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:06.241324 5060 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 14:11:06.243873 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.243830 5060 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 14:11:06.244183 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.244164 5060 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 14:11:06.253689 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.253656 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/1.log" Nov 03 14:11:06.253776 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.253701 5060 generic.go:356] "Generic (PLEG): container finished" podID="307e48f6-4a66-4c9c-a4ff-848cce9d3e0e" containerID="d87a44a09d0ad13a61108d4537be53ab5164f5519ade849da1c1ba1e459d0490" exitCode=255 Nov 03 14:11:06.257286 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.257267 5060 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 14:11:06.266197 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.266175 5060 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 14:11:06.294123 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.294091 5060 generic.go:356] "Generic (PLEG): container finished" podID="1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd" containerID="dc902db5787a87cb4794c1abafe07a682e636cc03ded505a54b6f1cd45fba344" exitCode=0 Nov 03 14:11:06.294123 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.294124 5060 generic.go:356] "Generic (PLEG): container finished" podID="1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd" containerID="7e38f3de7779fc4b5765c2a52708d60c0655f5782980305ce78e40a72a718bbc" exitCode=0 Nov 03 14:11:06.294283 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.294137 5060 generic.go:356] "Generic (PLEG): container finished" podID="1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd" containerID="0d36a983a6dc99c9069dc3bcd0b0a738e8c47b13de3419ee1eb1188f6a060ec2" exitCode=0 Nov 03 14:11:06.294283 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.294150 5060 generic.go:356] "Generic (PLEG): container finished" podID="1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd" containerID="0dc793aba877130196fa01ab8f611e814d8d270ade124a55f7fbc073c84a6ad0" exitCode=0 Nov 03 14:11:06.294283 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.294159 5060 generic.go:356] "Generic (PLEG): container finished" podID="1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd" containerID="5bdad1c802cf9cc8998f00321668b0c4cfc118fda6adf2e0ea6f2dac14495361" exitCode=0 Nov 03 14:11:06.294283 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.294171 5060 generic.go:356] "Generic (PLEG): container finished" podID="1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd" containerID="27776870886ef7404c376ae0dfcfb80212a9ca04bee79ea22da0db38709d2d93" exitCode=0 Nov 03 14:11:06.301731 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.301702 5060 generic.go:356] "Generic (PLEG): container finished" podID="84a79a081fb234a3eba8ea0d2c88476d" containerID="1c22cb15762f0f547cf6c6ceefb39cd850e4948f17aa3fe53bc665f93e6897af" exitCode=0 Nov 03 14:11:06.307799 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.307778 5060 manager.go:324] Recovery completed Nov 03 14:11:06.328912 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.328860 5060 cpu_manager.go:226] "Starting" policy="none" Nov 03 14:11:06.328912 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.328876 5060 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 14:11:06.328912 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.328897 5060 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 14:11:06.329052 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.329036 5060 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 14:11:06.329084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.329054 5060 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 14:11:06.329084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.329077 5060 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 14:11:06.329161 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.329084 5060 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 14:11:06.329161 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.329092 5060 policy_none.go:50] "Start" Nov 03 14:11:06.329161 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.329105 5060 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 14:11:06.329161 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.329119 5060 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 14:11:06.342113 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:06.342090 5060 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 03 14:11:06.347657 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.347640 5060 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 14:11:06.347657 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.347658 5060 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 14:11:06.347771 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.347665 5060 policy_none.go:44] "Start" Nov 03 14:11:06.353566 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.353535 5060 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 14:11:06.356070 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.356048 5060 manager.go:341] "Starting Device Plugin manager" Nov 03 14:11:06.356544 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:06.356079 5060 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 14:11:06.356585 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.356546 5060 server.go:87] "Starting device plugin registration server" Nov 03 14:11:06.356712 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.356703 5060 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 14:11:06.356742 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.356714 5060 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 14:11:06.356826 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.356809 5060 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 14:11:06.356973 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.356933 5060 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 14:11:06.357065 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.356984 5060 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 14:11:06.357065 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.356990 5060 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 14:11:06.357175 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.357039 5060 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T14:11:06.356968183Z","UUID":"ffd2c105-64f0-4733-8255-343029ca567e","Handler":null,"Name":"","Endpoint":""} Nov 03 14:11:06.358297 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:06.358273 5060 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 14:11:06.358392 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.358277 5060 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 14:11:06.358392 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.358383 5060 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 14:11:06.457336 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.457307 5060 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 14:11:06.458639 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.458620 5060 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-171.ec2.internal" event="NodeHasSufficientMemory" Nov 03 14:11:06.458736 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.458652 5060 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-171.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 14:11:06.458736 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.458662 5060 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-171.ec2.internal" event="NodeHasSufficientPID" Nov 03 14:11:06.458736 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.458702 5060 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-171.ec2.internal" Nov 03 14:11:06.512132 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.512104 5060 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-143-171.ec2.internal" Nov 03 14:11:06.512300 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.512190 5060 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-171.ec2.internal" Nov 03 14:11:06.542577 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.542526 5060 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-171.ec2.internal"] Nov 03 14:11:06.543339 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.543289 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-171.ec2.internal" event={"ID":"3a1b8625ac3f3be5fcd1bbb72178f5c6","Type":"ContainerStarted","Data":"d03de8744420466b622bbc39eaaa70195f847bcc958bbc09ca60040ef30d51b0"} Nov 03 14:11:06.543455 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.543351 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-171.ec2.internal" event={"ID":"3a1b8625ac3f3be5fcd1bbb72178f5c6","Type":"ContainerStarted","Data":"ddafe78a5f8133c3e0127472345a985b2211323b1eefc6f97145d0d119c24cd2"} Nov 03 14:11:06.543455 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.543416 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal" event={"ID":"84a79a081fb234a3eba8ea0d2c88476d","Type":"ContainerStarted","Data":"c4f9f990415b112f9c4b71851c65c694ff22433c825396f5deb00e5d82cef063"} Nov 03 14:11:06.543455 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.543352 5060 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-171.ec2.internal" Nov 03 14:11:06.543552 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.543481 5060 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal" Nov 03 14:11:06.543584 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.543434 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal" event={"ID":"84a79a081fb234a3eba8ea0d2c88476d","Type":"ContainerDied","Data":"1c22cb15762f0f547cf6c6ceefb39cd850e4948f17aa3fe53bc665f93e6897af"} Nov 03 14:11:06.543613 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.543584 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal" event={"ID":"84a79a081fb234a3eba8ea0d2c88476d","Type":"ContainerStarted","Data":"f05d75fa7171d422ee7851daa8ca8d3543fdf8ecc1043d80c30e26d7ef0e46de"} Nov 03 14:11:06.575981 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.575956 5060 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 14:11:06.576089 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:06.576004 5060 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-171.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-171.ec2.internal" Nov 03 14:11:06.577284 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.577267 5060 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 14:11:06.577363 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:06.577306 5060 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal" Nov 03 14:11:06.643426 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.643399 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/84a79a081fb234a3eba8ea0d2c88476d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal\" (UID: \"84a79a081fb234a3eba8ea0d2c88476d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal" Nov 03 14:11:06.643511 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.643442 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/84a79a081fb234a3eba8ea0d2c88476d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal\" (UID: \"84a79a081fb234a3eba8ea0d2c88476d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal" Nov 03 14:11:06.643511 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.643464 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3a1b8625ac3f3be5fcd1bbb72178f5c6-config\") pod \"kube-apiserver-proxy-ip-10-0-143-171.ec2.internal\" (UID: \"3a1b8625ac3f3be5fcd1bbb72178f5c6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-171.ec2.internal" Nov 03 14:11:06.744123 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.744095 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3a1b8625ac3f3be5fcd1bbb72178f5c6-config\") pod \"kube-apiserver-proxy-ip-10-0-143-171.ec2.internal\" (UID: \"3a1b8625ac3f3be5fcd1bbb72178f5c6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-171.ec2.internal" Nov 03 14:11:06.744123 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.744112 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3a1b8625ac3f3be5fcd1bbb72178f5c6-config\") pod \"kube-apiserver-proxy-ip-10-0-143-171.ec2.internal\" (UID: \"3a1b8625ac3f3be5fcd1bbb72178f5c6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-171.ec2.internal" Nov 03 14:11:06.744274 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.744137 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/84a79a081fb234a3eba8ea0d2c88476d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal\" (UID: \"84a79a081fb234a3eba8ea0d2c88476d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal" Nov 03 14:11:06.744274 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.744162 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/84a79a081fb234a3eba8ea0d2c88476d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal\" (UID: \"84a79a081fb234a3eba8ea0d2c88476d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal" Nov 03 14:11:06.744274 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.744220 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/84a79a081fb234a3eba8ea0d2c88476d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal\" (UID: \"84a79a081fb234a3eba8ea0d2c88476d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal" Nov 03 14:11:06.744274 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:06.744232 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/84a79a081fb234a3eba8ea0d2c88476d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal\" (UID: \"84a79a081fb234a3eba8ea0d2c88476d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal" Nov 03 14:11:07.176039 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.176003 5060 apiserver.go:52] "Watching apiserver" Nov 03 14:11:07.247234 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.247198 5060 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 14:11:07.248005 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.247971 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-pn8kf","openshift-multus/multus-q5cqm","openshift-service-ca-operator/service-ca-operator-544d749c8c-nvj5v","openshift-insights/insights-operator-7c4986d449-j66qt","openshift-dns/dns-default-2bk97","openshift-network-console/networking-console-plugin-86c478bb88-zgd4k","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal","openshift-insights/insights-runtime-extractor-sm2jg","openshift-network-diagnostics/network-check-source-6f78d59f99-rjz49","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-slvhl","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-v9mxv","openshift-image-registry/image-registry-57647df665-9q52v","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q79vl","kube-system/kas-connection-checker-685895c7c9-g28dw","openshift-ovn-kubernetes/ovnkube-node-w6d6z","openshift-kube-storage-version-migrator/migrator-66df5d496b-c8hlq","openshift-service-ca/service-ca-5fffc97bdf-9vrkp","openshift-insights/periodic-gathering-fzk44-mlg6z","openshift-ingress/router-default-6c7f54c9b7-t6wbf","kube-system/kube-apiserver-proxy-ip-10-0-143-171.ec2.internal","kube-system/global-pull-secret-syncer-2xrr4","openshift-image-registry/node-ca-t4b94","openshift-monitoring/cluster-monitoring-operator-884576c57-v6pjk","openshift-dns/node-resolver-q4gsb","kube-system/kas-connection-checker-685895c7c9-wcb9f","openshift-network-operator/iptables-alerter-65nrx","openshift-image-registry/image-registry-6d7c596f48-qrcsg","openshift-multus/multus-additional-cni-plugins-kfkzp","openshift-ingress-canary/ingress-canary-qnjdm","kube-system/konnectivity-agent-dj8cg","openshift-multus/network-metrics-daemon-tc5nw","openshift-cluster-node-tuning-operator/tuned-xcplz","kube-system/kas-connection-checker-685895c7c9-pkhg9","openshift-network-diagnostics/network-check-target-xkhr9","openshift-console-operator/console-operator-7cd56b8dbc-5xzwg"] Nov 03 14:11:07.249933 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.249908 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v6pjk" Nov 03 14:11:07.250375 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.250359 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:11:07.252675 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.252656 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:11:07.253117 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.253102 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q79vl" Nov 03 14:11:07.253501 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.253485 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-zgd4k" Nov 03 14:11:07.257894 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.257875 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc5nw" Nov 03 14:11:07.261950 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.261929 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:11:07.263215 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.263199 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2bk97" Nov 03 14:11:07.264650 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.264634 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 14:11:07.264721 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.264637 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 14:11:07.265167 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.265150 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 14:11:07.265250 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.265183 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 14:11:07.265250 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.265204 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 14:11:07.265324 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.265252 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 14:11:07.265769 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.265747 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 14:11:07.265921 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.265902 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:07.266068 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.266047 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 14:11:07.266132 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.266061 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 14:11:07.278386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.278367 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 14:11:07.278549 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.278533 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 14:11:07.279352 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.279335 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 14:11:07.280507 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.280485 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 14:11:07.280590 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.280546 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 14:11:07.281601 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.281579 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 14:11:07.281812 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.281789 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 14:11:07.282105 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.282087 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 14:11:07.282675 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.282654 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 14:11:07.282801 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.282785 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 14:11:07.282941 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.282926 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 14:11:07.282997 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.282933 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 14:11:07.283165 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.283150 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 14:11:07.283213 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.283195 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 14:11:07.283259 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.283245 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 14:11:07.283440 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.283418 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 14:11:07.283498 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.283451 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 14:11:07.283735 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.283714 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 14:11:07.283790 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.283769 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 14:11:07.283884 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.283867 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 14:11:07.283943 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.283909 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 14:11:07.284077 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.284045 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 14:11:07.284232 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.284180 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 14:11:07.284232 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.284189 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 14:11:07.284341 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.284305 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xpq6s\"" Nov 03 14:11:07.284467 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.284444 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 14:11:07.284531 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.284473 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 14:11:07.284531 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.284479 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 14:11:07.284642 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.284627 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 14:11:07.284714 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.284652 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 14:11:07.284819 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.284805 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7xcd5\"" Nov 03 14:11:07.284948 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.284805 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 14:11:07.284948 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.284869 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 14:11:07.284948 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.284934 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 14:11:07.285165 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.284950 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 14:11:07.285165 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.285019 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 14:11:07.285165 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.285155 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 14:11:07.285318 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.285253 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 14:11:07.285318 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.285162 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 14:11:07.285318 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.285264 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 14:11:07.285461 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.285335 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 14:11:07.285461 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.285380 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 14:11:07.285461 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.285442 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-nr7b5\"" Nov 03 14:11:07.285461 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.285471 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 14:11:07.285694 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.285607 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 14:11:07.285815 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.285778 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-fxpd5\"" Nov 03 14:11:07.285919 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.285824 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 14:11:07.285961 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.285937 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 14:11:07.286013 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.285988 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 14:11:07.286060 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.286029 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 14:11:07.286060 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.286034 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 14:11:07.286060 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.286058 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 14:11:07.286160 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.286074 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 14:11:07.286753 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.286739 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 14:11:07.286811 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.286760 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 14:11:07.286811 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.286773 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 14:11:07.286811 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.286784 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-jtpd7\"" Nov 03 14:11:07.286811 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.286797 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 14:11:07.287012 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.286829 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 14:11:07.287012 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.286830 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 14:11:07.287012 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.286884 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qgxkz\"" Nov 03 14:11:07.287012 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.286908 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-bgwlk\"" Nov 03 14:11:07.287012 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.286975 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 14:11:07.287012 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.287011 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 14:11:07.287278 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.287055 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 14:11:07.287278 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.287071 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 14:11:07.287278 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.287098 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 14:11:07.287278 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.287141 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2ft6v\"" Nov 03 14:11:07.287278 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.287190 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 14:11:07.287278 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.287209 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rmj5t\"" Nov 03 14:11:07.291515 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.291500 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-lqmd6\"" Nov 03 14:11:07.291515 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.291501 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-q495v\"" Nov 03 14:11:07.291666 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.291529 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 14:11:07.291666 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.291572 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 14:11:07.291782 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.291767 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xm8pf\"" Nov 03 14:11:07.291854 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.291787 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-95wl7\"" Nov 03 14:11:07.291854 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.291817 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qhrll\"" Nov 03 14:11:07.291854 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.291830 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 14:11:07.291854 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.291828 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 14:11:07.292072 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.291857 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-r6xhc\"" Nov 03 14:11:07.292072 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.291891 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-tb44c\"" Nov 03 14:11:07.292072 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.291906 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 14:11:07.292072 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.291948 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 14:11:07.292072 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.291969 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ghsgd\"" Nov 03 14:11:07.292072 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.292053 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-trbj6\"" Nov 03 14:11:07.292398 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.292388 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 14:11:07.292461 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.292427 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ml487\"" Nov 03 14:11:07.292586 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.292568 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 14:11:07.292623 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.292612 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-qgp8d\"" Nov 03 14:11:07.292651 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.292619 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-h44zq\"" Nov 03 14:11:07.292737 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.292718 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 14:11:07.293365 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.293323 5060 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 14:11:07.293657 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.293599 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 14:11:07.293657 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.293603 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 14:11:07.293657 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.293643 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 14:11:07.293828 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.293671 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 14:11:07.293828 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.293672 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 14:11:07.293828 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.293692 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 14:11:07.293828 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.293674 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 14:11:07.293828 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.293716 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hlrhs\"" Nov 03 14:11:07.293828 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.293813 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-hhz2n\"" Nov 03 14:11:07.305508 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.305484 5060 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal" Nov 03 14:11:07.305618 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.305593 5060 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-171.ec2.internal" Nov 03 14:11:07.330354 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.330328 5060 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 14:11:07.330471 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:07.330386 5060 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-171.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-171.ec2.internal" Nov 03 14:11:07.331386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.331366 5060 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 14:11:07.331431 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:07.331409 5060 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-171.ec2.internal" Nov 03 14:11:07.346660 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.346630 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-image-registry-private-configuration\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:11:07.346660 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.346664 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef43449c-b7dc-4e84-b785-63201f80d4da-original-pull-secret\") pod \"global-pull-secret-syncer-2xrr4\" (UID: \"ef43449c-b7dc-4e84-b785-63201f80d4da\") " pod="kube-system/global-pull-secret-syncer-2xrr4" Nov 03 14:11:07.346949 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.346682 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fbfeab8a-b74e-4188-b565-b56442a01b79-installation-pull-secrets\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:11:07.346949 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.346713 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd-system-cni-dir\") pod \"multus-additional-cni-plugins-kfkzp\" (UID: \"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd\") " pod="openshift-multus/multus-additional-cni-plugins-kfkzp" Nov 03 14:11:07.346949 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.346738 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kfkzp\" (UID: \"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd\") " pod="openshift-multus/multus-additional-cni-plugins-kfkzp" Nov 03 14:11:07.346949 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.346793 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-sys\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:11:07.346949 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.346820 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bf085d57-4ae4-4add-9402-9ee1ce5d795a-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-slvhl\" (UID: \"bf085d57-4ae4-4add-9402-9ee1ce5d795a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-slvhl" Nov 03 14:11:07.346949 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.346863 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-etc-modprobe-d\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:11:07.346949 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.346888 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-multus-socket-dir-parent\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.346949 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.346909 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-host-run-k8s-cni-cncf-io\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.346949 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.346927 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-etc-kubernetes\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.347319 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.346977 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fbfeab8a-b74e-4188-b565-b56442a01b79-image-registry-private-configuration\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:11:07.347319 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.347010 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/396e7be7-0e72-4c32-a04a-22bd9bd708e1-iptables-alerter-script\") pod \"iptables-alerter-65nrx\" (UID: \"396e7be7-0e72-4c32-a04a-22bd9bd708e1\") " pod="openshift-network-operator/iptables-alerter-65nrx" Nov 03 14:11:07.347319 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.347038 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-etc-kubernetes\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:11:07.347319 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.347041 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fbfeab8a-b74e-4188-b565-b56442a01b79-installation-pull-secrets\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:11:07.347319 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.347045 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef43449c-b7dc-4e84-b785-63201f80d4da-original-pull-secret\") pod \"global-pull-secret-syncer-2xrr4\" (UID: \"ef43449c-b7dc-4e84-b785-63201f80d4da\") " pod="kube-system/global-pull-secret-syncer-2xrr4" Nov 03 14:11:07.347319 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.347032 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-image-registry-private-configuration\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:11:07.347319 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.347071 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bf085d57-4ae4-4add-9402-9ee1ce5d795a-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-slvhl\" (UID: \"bf085d57-4ae4-4add-9402-9ee1ce5d795a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-slvhl" Nov 03 14:11:07.347319 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.347087 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fbfeab8a-b74e-4188-b565-b56442a01b79-ca-trust-extracted\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:11:07.347319 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.347130 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ea7bdafa-8d46-4b28-b93b-b73352b2e269-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-q79vl\" (UID: \"ea7bdafa-8d46-4b28-b93b-b73352b2e269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q79vl" Nov 03 14:11:07.347319 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.347144 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fbfeab8a-b74e-4188-b565-b56442a01b79-ca-trust-extracted\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:11:07.347319 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.347156 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fbfeab8a-b74e-4188-b565-b56442a01b79-trusted-ca\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:11:07.347319 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.347171 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fbfeab8a-b74e-4188-b565-b56442a01b79-image-registry-private-configuration\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:11:07.347319 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.347181 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-installation-pull-secrets\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:11:07.347319 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.347204 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-run\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:11:07.347319 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.347225 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-host-run-netns\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.347319 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.347251 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-hostroot\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.347319 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.347296 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ea7bdafa-8d46-4b28-b93b-b73352b2e269-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-q79vl\" (UID: \"ea7bdafa-8d46-4b28-b93b-b73352b2e269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q79vl" Nov 03 14:11:07.347319 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.347312 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/396e7be7-0e72-4c32-a04a-22bd9bd708e1-iptables-alerter-script\") pod \"iptables-alerter-65nrx\" (UID: \"396e7be7-0e72-4c32-a04a-22bd9bd708e1\") " pod="openshift-network-operator/iptables-alerter-65nrx" Nov 03 14:11:07.348116 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.347334 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmwwz\" (UniqueName: \"kubernetes.io/projected/396e7be7-0e72-4c32-a04a-22bd9bd708e1-kube-api-access-fmwwz\") pod \"iptables-alerter-65nrx\" (UID: \"396e7be7-0e72-4c32-a04a-22bd9bd708e1\") " pod="openshift-network-operator/iptables-alerter-65nrx" Nov 03 14:11:07.348116 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.347359 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3fe332c5-8991-4d94-9b67-c9017f4945b8-cni-binary-copy\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.348116 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.347407 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/36065516-87ab-4057-b198-439305606d12-stats-auth\") pod \"router-default-6c7f54c9b7-t6wbf\" (UID: \"36065516-87ab-4057-b198-439305606d12\") " pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:11:07.348116 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.347413 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-installation-pull-secrets\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:11:07.348116 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.347414 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ea7bdafa-8d46-4b28-b93b-b73352b2e269-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-q79vl\" (UID: \"ea7bdafa-8d46-4b28-b93b-b73352b2e269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q79vl" Nov 03 14:11:07.348116 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.347435 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrqb5\" (UniqueName: \"kubernetes.io/projected/63bfb9d0-9e87-44c6-a6f7-2f7b992e20f6-kube-api-access-hrqb5\") pod \"ingress-canary-qnjdm\" (UID: \"63bfb9d0-9e87-44c6-a6f7-2f7b992e20f6\") " pod="openshift-ingress-canary/ingress-canary-qnjdm" Nov 03 14:11:07.348116 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.347487 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdggk\" (UniqueName: \"kubernetes.io/projected/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-kube-api-access-hdggk\") pod \"insights-runtime-extractor-sm2jg\" (UID: \"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee\") " pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:07.348116 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.347512 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-multus-conf-dir\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.348116 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.347535 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfv25\" (UniqueName: \"kubernetes.io/projected/d533df4c-38b2-42b2-9cb2-b82afac33795-kube-api-access-sfv25\") pod \"aws-ebs-csi-driver-node-zkvrz\" (UID: \"d533df4c-38b2-42b2-9cb2-b82afac33795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:11:07.348116 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.347565 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkjrd\" (UniqueName: \"kubernetes.io/projected/328dde79-0c85-47a3-b3f8-38d8cc8614c0-kube-api-access-dkjrd\") pod \"network-check-target-xkhr9\" (UID: \"328dde79-0c85-47a3-b3f8-38d8cc8614c0\") " pod="openshift-network-diagnostics/network-check-target-xkhr9" Nov 03 14:11:07.348116 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.347565 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fbfeab8a-b74e-4188-b565-b56442a01b79-trusted-ca\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:11:07.348116 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.347599 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr7zc\" (UniqueName: \"kubernetes.io/projected/e637e650-5fc6-4adb-a97a-47d924c30f97-kube-api-access-mr7zc\") pod \"node-resolver-q4gsb\" (UID: \"e637e650-5fc6-4adb-a97a-47d924c30f97\") " pod="openshift-dns/node-resolver-q4gsb" Nov 03 14:11:07.348116 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.347649 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/36065516-87ab-4057-b198-439305606d12-stats-auth\") pod \"router-default-6c7f54c9b7-t6wbf\" (UID: \"36065516-87ab-4057-b198-439305606d12\") " pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:11:07.348116 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.347685 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/36065516-87ab-4057-b198-439305606d12-default-certificate\") pod \"router-default-6c7f54c9b7-t6wbf\" (UID: \"36065516-87ab-4057-b198-439305606d12\") " pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:11:07.348116 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.347781 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5m26\" (UniqueName: \"kubernetes.io/projected/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d-kube-api-access-m5m26\") pod \"periodic-gathering-fzk44-mlg6z\" (UID: \"7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d\") " pod="openshift-insights/periodic-gathering-fzk44-mlg6z" Nov 03 14:11:07.348116 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.347821 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3fe332c5-8991-4d94-9b67-c9017f4945b8-cni-binary-copy\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.348116 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.347876 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kfkzp\" (UID: \"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd\") " pod="openshift-multus/multus-additional-cni-plugins-kfkzp" Nov 03 14:11:07.348116 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.347916 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.348116 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.347949 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd3829b5-04dc-4112-b2d9-087d136ba9ec-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-j66qt\" (UID: \"dd3829b5-04dc-4112-b2d9-087d136ba9ec\") " pod="openshift-insights/insights-operator-7c4986d449-j66qt" Nov 03 14:11:07.348116 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.347981 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/36065516-87ab-4057-b198-439305606d12-default-certificate\") pod \"router-default-6c7f54c9b7-t6wbf\" (UID: \"36065516-87ab-4057-b198-439305606d12\") " pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:11:07.348116 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.347977 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bdf183c4-a9ee-4ea8-ace1-50f307114328-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-v6pjk\" (UID: \"bdf183c4-a9ee-4ea8-ace1-50f307114328\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v6pjk" Nov 03 14:11:07.348116 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.348025 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwf5m\" (UniqueName: \"kubernetes.io/projected/8d74c5e4-f631-4eae-84d5-84749b53ccf8-kube-api-access-nwf5m\") pod \"service-ca-operator-544d749c8c-nvj5v\" (UID: \"8d74c5e4-f631-4eae-84d5-84749b53ccf8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nvj5v" Nov 03 14:11:07.348116 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.348084 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5qqj\" (UniqueName: \"kubernetes.io/projected/60d2d1a6-a89f-4a9a-acb9-e85ce2c8f6ce-kube-api-access-w5qqj\") pod \"migrator-66df5d496b-pn8kf\" (UID: \"60d2d1a6-a89f-4a9a-acb9-e85ce2c8f6ce\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pn8kf" Nov 03 14:11:07.348116 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.348110 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kfkzp\" (UID: \"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd\") " pod="openshift-multus/multus-additional-cni-plugins-kfkzp" Nov 03 14:11:07.348116 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.348109 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6466fc4-ed3d-4031-ba11-5606bd1af74b-tmp\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.348179 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6466fc4-ed3d-4031-ba11-5606bd1af74b-tmp\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.348161 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-registry-certificates\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.348233 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n295\" (UniqueName: \"kubernetes.io/projected/11d16990-ce0c-458f-ab1e-e9a3fed3221b-kube-api-access-8n295\") pod \"volume-data-source-validator-58c9f5f46d-v9mxv\" (UID: \"11d16990-ce0c-458f-ab1e-e9a3fed3221b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-v9mxv" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.348239 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bdf183c4-a9ee-4ea8-ace1-50f307114328-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-v6pjk\" (UID: \"bdf183c4-a9ee-4ea8-ace1-50f307114328\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v6pjk" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.348260 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd5tr\" (UniqueName: \"kubernetes.io/projected/47c859b2-a25b-45e5-acdc-0632d472c097-kube-api-access-pd5tr\") pod \"kas-connection-checker-685895c7c9-pkhg9\" (UID: \"47c859b2-a25b-45e5-acdc-0632d472c097\") " pod="kube-system/kas-connection-checker-685895c7c9-pkhg9" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.348263 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd3829b5-04dc-4112-b2d9-087d136ba9ec-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-j66qt\" (UID: \"dd3829b5-04dc-4112-b2d9-087d136ba9ec\") " pod="openshift-insights/insights-operator-7c4986d449-j66qt" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.348290 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-bound-sa-token\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.348315 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-data-volume\") pod \"insights-runtime-extractor-sm2jg\" (UID: \"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee\") " pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.348348 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-626zm\" (UniqueName: \"kubernetes.io/projected/840ddc67-db99-4d3f-ba2d-51632ac3912c-kube-api-access-626zm\") pod \"network-check-source-6f78d59f99-rjz49\" (UID: \"840ddc67-db99-4d3f-ba2d-51632ac3912c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rjz49" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.348373 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/61060804-c519-490a-ac90-674792d50174-config-volume\") pod \"dns-default-2bk97\" (UID: \"61060804-c519-490a-ac90-674792d50174\") " pod="openshift-dns/dns-default-2bk97" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.348395 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-data-volume\") pod \"insights-runtime-extractor-sm2jg\" (UID: \"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee\") " pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.348401 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d533df4c-38b2-42b2-9cb2-b82afac33795-registration-dir\") pod \"aws-ebs-csi-driver-node-zkvrz\" (UID: \"d533df4c-38b2-42b2-9cb2-b82afac33795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.348332 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-registry-certificates\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.348489 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkrl8\" (UniqueName: \"kubernetes.io/projected/f52cae01-7951-4d9e-9a57-9ba5bcdf6857-kube-api-access-fkrl8\") pod \"network-metrics-daemon-tc5nw\" (UID: \"f52cae01-7951-4d9e-9a57-9ba5bcdf6857\") " pod="openshift-multus/network-metrics-daemon-tc5nw" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.348518 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbw2q\" (UniqueName: \"kubernetes.io/projected/bf085d57-4ae4-4add-9402-9ee1ce5d795a-kube-api-access-pbw2q\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-slvhl\" (UID: \"bf085d57-4ae4-4add-9402-9ee1ce5d795a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-slvhl" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.348540 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-host-cni-netd\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.348589 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/61060804-c519-490a-ac90-674792d50174-config-volume\") pod \"dns-default-2bk97\" (UID: \"61060804-c519-490a-ac90-674792d50174\") " pod="openshift-dns/dns-default-2bk97" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.348594 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-ovn-node-metrics-cert\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.348638 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd3829b5-04dc-4112-b2d9-087d136ba9ec-tmp\") pod \"insights-operator-7c4986d449-j66qt\" (UID: \"dd3829b5-04dc-4112-b2d9-087d136ba9ec\") " pod="openshift-insights/insights-operator-7c4986d449-j66qt" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.348677 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/dd3829b5-04dc-4112-b2d9-087d136ba9ec-snapshots\") pod \"insights-operator-7c4986d449-j66qt\" (UID: \"dd3829b5-04dc-4112-b2d9-087d136ba9ec\") " pod="openshift-insights/insights-operator-7c4986d449-j66qt" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.348713 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d-trusted-ca-bundle\") pod \"periodic-gathering-fzk44-mlg6z\" (UID: \"7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d\") " pod="openshift-insights/periodic-gathering-fzk44-mlg6z" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.348733 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-etc-sysctl-conf\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.348733 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/dd3829b5-04dc-4112-b2d9-087d136ba9ec-snapshots\") pod \"insights-operator-7c4986d449-j66qt\" (UID: \"dd3829b5-04dc-4112-b2d9-087d136ba9ec\") " pod="openshift-insights/insights-operator-7c4986d449-j66qt" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.348713 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd3829b5-04dc-4112-b2d9-087d136ba9ec-tmp\") pod \"insights-operator-7c4986d449-j66qt\" (UID: \"dd3829b5-04dc-4112-b2d9-087d136ba9ec\") " pod="openshift-insights/insights-operator-7c4986d449-j66qt" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.348763 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-var-lib-kubelet\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.348782 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sm2jg\" (UID: \"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee\") " pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.348804 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-cnibin\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.348825 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3fe332c5-8991-4d94-9b67-c9017f4945b8-multus-daemon-config\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.348904 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd-cnibin\") pod \"multus-additional-cni-plugins-kfkzp\" (UID: \"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd\") " pod="openshift-multus/multus-additional-cni-plugins-kfkzp" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.348923 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-ovn-node-metrics-cert\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.348967 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dd3829b5-04dc-4112-b2d9-087d136ba9ec-serving-cert\") pod \"insights-operator-7c4986d449-j66qt\" (UID: \"dd3829b5-04dc-4112-b2d9-087d136ba9ec\") " pod="openshift-insights/insights-operator-7c4986d449-j66qt" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.348996 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d-trusted-ca-bundle\") pod \"periodic-gathering-fzk44-mlg6z\" (UID: \"7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d\") " pod="openshift-insights/periodic-gathering-fzk44-mlg6z" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.349066 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3fe332c5-8991-4d94-9b67-c9017f4945b8-multus-daemon-config\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.349115 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dd3829b5-04dc-4112-b2d9-087d136ba9ec-serving-cert\") pod \"insights-operator-7c4986d449-j66qt\" (UID: \"dd3829b5-04dc-4112-b2d9-087d136ba9ec\") " pod="openshift-insights/insights-operator-7c4986d449-j66qt" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.349123 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/32d5aaf4-3430-4e5b-aca5-f0d720ce8657-signing-key\") pod \"service-ca-5fffc97bdf-9vrkp\" (UID: \"32d5aaf4-3430-4e5b-aca5-f0d720ce8657\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9vrkp" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.349165 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k4jz\" (UniqueName: \"kubernetes.io/projected/36065516-87ab-4057-b198-439305606d12-kube-api-access-8k4jz\") pod \"router-default-6c7f54c9b7-t6wbf\" (UID: \"36065516-87ab-4057-b198-439305606d12\") " pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.349220 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d-service-ca-bundle\") pod \"periodic-gathering-fzk44-mlg6z\" (UID: \"7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d\") " pod="openshift-insights/periodic-gathering-fzk44-mlg6z" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.349292 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/32d5aaf4-3430-4e5b-aca5-f0d720ce8657-signing-key\") pod \"service-ca-5fffc97bdf-9vrkp\" (UID: \"32d5aaf4-3430-4e5b-aca5-f0d720ce8657\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9vrkp" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.349319 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/307e48f6-4a66-4c9c-a4ff-848cce9d3e0e-config\") pod \"console-operator-7cd56b8dbc-5xzwg\" (UID: \"307e48f6-4a66-4c9c-a4ff-848cce9d3e0e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.349336 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr24x\" (UniqueName: \"kubernetes.io/projected/1da80d21-c62b-44ec-a3fe-64633b5de401-kube-api-access-nr24x\") pod \"migrator-66df5d496b-c8hlq\" (UID: \"1da80d21-c62b-44ec-a3fe-64633b5de401\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-c8hlq" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.349359 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36065516-87ab-4057-b198-439305606d12-service-ca-bundle\") pod \"router-default-6c7f54c9b7-t6wbf\" (UID: \"36065516-87ab-4057-b198-439305606d12\") " pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.349402 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bdf183c4-a9ee-4ea8-ace1-50f307114328-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-v6pjk\" (UID: \"bdf183c4-a9ee-4ea8-ace1-50f307114328\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v6pjk" Nov 03 14:11:07.349386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.349428 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtr9w\" (UniqueName: \"kubernetes.io/projected/bdf183c4-a9ee-4ea8-ace1-50f307114328-kube-api-access-vtr9w\") pod \"cluster-monitoring-operator-884576c57-v6pjk\" (UID: \"bdf183c4-a9ee-4ea8-ace1-50f307114328\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v6pjk" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.349474 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4w2t\" (UniqueName: \"kubernetes.io/projected/61060804-c519-490a-ac90-674792d50174-kube-api-access-h4w2t\") pod \"dns-default-2bk97\" (UID: \"61060804-c519-490a-ac90-674792d50174\") " pod="openshift-dns/dns-default-2bk97" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.349484 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/307e48f6-4a66-4c9c-a4ff-848cce9d3e0e-config\") pod \"console-operator-7cd56b8dbc-5xzwg\" (UID: \"307e48f6-4a66-4c9c-a4ff-848cce9d3e0e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.349533 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-registry-tls\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.349590 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d533df4c-38b2-42b2-9cb2-b82afac33795-etc-selinux\") pod \"aws-ebs-csi-driver-node-zkvrz\" (UID: \"d533df4c-38b2-42b2-9cb2-b82afac33795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.349608 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d533df4c-38b2-42b2-9cb2-b82afac33795-sys-fs\") pod \"aws-ebs-csi-driver-node-zkvrz\" (UID: \"d533df4c-38b2-42b2-9cb2-b82afac33795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.349639 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-host-run-ovn-kubernetes\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.349667 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6f219713-6fd8-4580-b1c0-24e2a929c258-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zgd4k\" (UID: \"6f219713-6fd8-4580-b1c0-24e2a929c258\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zgd4k" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.349721 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-bound-sa-token\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.349752 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kfkzp\" (UID: \"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd\") " pod="openshift-multus/multus-additional-cni-plugins-kfkzp" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.349779 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbt8t\" (UniqueName: \"kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-kube-api-access-fbt8t\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.349805 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8d74c5e4-f631-4eae-84d5-84749b53ccf8-serving-cert\") pod \"service-ca-operator-544d749c8c-nvj5v\" (UID: \"8d74c5e4-f631-4eae-84d5-84749b53ccf8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nvj5v" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.349855 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d-archives-path\") pod \"periodic-gathering-fzk44-mlg6z\" (UID: \"7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d\") " pod="openshift-insights/periodic-gathering-fzk44-mlg6z" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.349867 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6f219713-6fd8-4580-b1c0-24e2a929c258-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zgd4k\" (UID: \"6f219713-6fd8-4580-b1c0-24e2a929c258\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zgd4k" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.349884 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pp5r\" (UniqueName: \"kubernetes.io/projected/00a412a1-3764-404b-9e6d-3a2bbb200781-kube-api-access-9pp5r\") pod \"kas-connection-checker-685895c7c9-g28dw\" (UID: \"00a412a1-3764-404b-9e6d-3a2bbb200781\") " pod="kube-system/kas-connection-checker-685895c7c9-g28dw" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.349919 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d-service-ca-bundle\") pod \"periodic-gathering-fzk44-mlg6z\" (UID: \"7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d\") " pod="openshift-insights/periodic-gathering-fzk44-mlg6z" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.349938 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d-archives-path\") pod \"periodic-gathering-fzk44-mlg6z\" (UID: \"7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d\") " pod="openshift-insights/periodic-gathering-fzk44-mlg6z" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.349963 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kfkzp\" (UID: \"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd\") " pod="openshift-multus/multus-additional-cni-plugins-kfkzp" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.349968 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-ca-trust-extracted\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.350012 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-ovnkube-config\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.350018 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-ca-trust-extracted\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.350023 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8d74c5e4-f631-4eae-84d5-84749b53ccf8-serving-cert\") pod \"service-ca-operator-544d749c8c-nvj5v\" (UID: \"8d74c5e4-f631-4eae-84d5-84749b53ccf8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nvj5v" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.350044 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd3829b5-04dc-4112-b2d9-087d136ba9ec-service-ca-bundle\") pod \"insights-operator-7c4986d449-j66qt\" (UID: \"dd3829b5-04dc-4112-b2d9-087d136ba9ec\") " pod="openshift-insights/insights-operator-7c4986d449-j66qt" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.350086 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-system-cni-dir\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.350103 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ef43449c-b7dc-4e84-b785-63201f80d4da-kubelet-config\") pod \"global-pull-secret-syncer-2xrr4\" (UID: \"ef43449c-b7dc-4e84-b785-63201f80d4da\") " pod="kube-system/global-pull-secret-syncer-2xrr4" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.350120 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/307e48f6-4a66-4c9c-a4ff-848cce9d3e0e-serving-cert\") pod \"console-operator-7cd56b8dbc-5xzwg\" (UID: \"307e48f6-4a66-4c9c-a4ff-848cce9d3e0e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.350184 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-ovnkube-config\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.350250 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-systemd-units\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.350259 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/307e48f6-4a66-4c9c-a4ff-848cce9d3e0e-serving-cert\") pod \"console-operator-7cd56b8dbc-5xzwg\" (UID: \"307e48f6-4a66-4c9c-a4ff-848cce9d3e0e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.350283 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a6466fc4-ed3d-4031-ba11-5606bd1af74b-etc-tuned\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.350308 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-multus-cni-dir\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.350346 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a6466fc4-ed3d-4031-ba11-5606bd1af74b-etc-tuned\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.350346 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-host-var-lib-cni-bin\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.350381 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-host-var-lib-kubelet\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.350398 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d533df4c-38b2-42b2-9cb2-b82afac33795-socket-dir\") pod \"aws-ebs-csi-driver-node-zkvrz\" (UID: \"d533df4c-38b2-42b2-9cb2-b82afac33795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.350413 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f52cae01-7951-4d9e-9a57-9ba5bcdf6857-metrics-certs\") pod \"network-metrics-daemon-tc5nw\" (UID: \"f52cae01-7951-4d9e-9a57-9ba5bcdf6857\") " pod="openshift-multus/network-metrics-daemon-tc5nw" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.350431 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61060804-c519-490a-ac90-674792d50174-tmp-dir\") pod \"dns-default-2bk97\" (UID: \"61060804-c519-490a-ac90-674792d50174\") " pod="openshift-dns/dns-default-2bk97" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.350462 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd3829b5-04dc-4112-b2d9-087d136ba9ec-service-ca-bundle\") pod \"insights-operator-7c4986d449-j66qt\" (UID: \"dd3829b5-04dc-4112-b2d9-087d136ba9ec\") " pod="openshift-insights/insights-operator-7c4986d449-j66qt" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.350479 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vktv4\" (UniqueName: \"kubernetes.io/projected/4d66aab2-4671-4bbd-a4cc-b9dd9c5a766d-kube-api-access-vktv4\") pod \"kas-connection-checker-685895c7c9-wcb9f\" (UID: \"4d66aab2-4671-4bbd-a4cc-b9dd9c5a766d\") " pod="kube-system/kas-connection-checker-685895c7c9-wcb9f" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.350486 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61060804-c519-490a-ac90-674792d50174-tmp-dir\") pod \"dns-default-2bk97\" (UID: \"61060804-c519-490a-ac90-674792d50174\") " pod="openshift-dns/dns-default-2bk97" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.350607 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-env-overrides\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.350646 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7782\" (UniqueName: \"kubernetes.io/projected/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-kube-api-access-g7782\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.350696 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-etc-sysconfig\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.350739 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-env-overrides\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.350791 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-etc-sysctl-d\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.350808 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e68be0c4-d4ee-4da5-82e3-82372f811841-host\") pod \"node-ca-t4b94\" (UID: \"e68be0c4-d4ee-4da5-82e3-82372f811841\") " pod="openshift-image-registry/node-ca-t4b94" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.350859 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgm54\" (UniqueName: \"kubernetes.io/projected/e68be0c4-d4ee-4da5-82e3-82372f811841-kube-api-access-wgm54\") pod \"node-ca-t4b94\" (UID: \"e68be0c4-d4ee-4da5-82e3-82372f811841\") " pod="openshift-image-registry/node-ca-t4b94" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.350898 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/396e7be7-0e72-4c32-a04a-22bd9bd708e1-host-slash\") pod \"iptables-alerter-65nrx\" (UID: \"396e7be7-0e72-4c32-a04a-22bd9bd708e1\") " pod="openshift-network-operator/iptables-alerter-65nrx" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.351004 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/307e48f6-4a66-4c9c-a4ff-848cce9d3e0e-trusted-ca\") pod \"console-operator-7cd56b8dbc-5xzwg\" (UID: \"307e48f6-4a66-4c9c-a4ff-848cce9d3e0e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.351034 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-run-systemd\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.351052 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-run-openvswitch\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.351088 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-run-ovn\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.351113 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-host-run-multus-certs\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.351142 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e68be0c4-d4ee-4da5-82e3-82372f811841-serviceca\") pod \"node-ca-t4b94\" (UID: \"e68be0c4-d4ee-4da5-82e3-82372f811841\") " pod="openshift-image-registry/node-ca-t4b94" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.351197 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp5h5\" (UniqueName: \"kubernetes.io/projected/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd-kube-api-access-vp5h5\") pod \"multus-additional-cni-plugins-kfkzp\" (UID: \"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd\") " pod="openshift-multus/multus-additional-cni-plugins-kfkzp" Nov 03 14:11:07.351217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.351254 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-log-socket\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.351322 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/307e48f6-4a66-4c9c-a4ff-848cce9d3e0e-trusted-ca\") pod \"console-operator-7cd56b8dbc-5xzwg\" (UID: \"307e48f6-4a66-4c9c-a4ff-848cce9d3e0e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.351369 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/32d5aaf4-3430-4e5b-aca5-f0d720ce8657-signing-cabundle\") pod \"service-ca-5fffc97bdf-9vrkp\" (UID: \"32d5aaf4-3430-4e5b-aca5-f0d720ce8657\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9vrkp" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.351384 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e68be0c4-d4ee-4da5-82e3-82372f811841-serviceca\") pod \"node-ca-t4b94\" (UID: \"e68be0c4-d4ee-4da5-82e3-82372f811841\") " pod="openshift-image-registry/node-ca-t4b94" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.351397 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36065516-87ab-4057-b198-439305606d12-metrics-certs\") pod \"router-default-6c7f54c9b7-t6wbf\" (UID: \"36065516-87ab-4057-b198-439305606d12\") " pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.351478 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e9dee2ef-6437-4963-8e27-da5fb9c75928-konnectivity-ca\") pod \"konnectivity-agent-dj8cg\" (UID: \"e9dee2ef-6437-4963-8e27-da5fb9c75928\") " pod="kube-system/konnectivity-agent-dj8cg" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.351511 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63bfb9d0-9e87-44c6-a6f7-2f7b992e20f6-cert\") pod \"ingress-canary-qnjdm\" (UID: \"63bfb9d0-9e87-44c6-a6f7-2f7b992e20f6\") " pod="openshift-ingress-canary/ingress-canary-qnjdm" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.351531 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/32d5aaf4-3430-4e5b-aca5-f0d720ce8657-signing-cabundle\") pod \"service-ca-5fffc97bdf-9vrkp\" (UID: \"32d5aaf4-3430-4e5b-aca5-f0d720ce8657\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9vrkp" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.351561 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-host-slash\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.351662 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-ovnkube-script-lib\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.351704 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-host\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.351722 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63bfb9d0-9e87-44c6-a6f7-2f7b992e20f6-cert\") pod \"ingress-canary-qnjdm\" (UID: \"63bfb9d0-9e87-44c6-a6f7-2f7b992e20f6\") " pod="openshift-ingress-canary/ingress-canary-qnjdm" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.351739 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-trusted-ca\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.351749 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e9dee2ef-6437-4963-8e27-da5fb9c75928-konnectivity-ca\") pod \"konnectivity-agent-dj8cg\" (UID: \"e9dee2ef-6437-4963-8e27-da5fb9c75928\") " pod="kube-system/konnectivity-agent-dj8cg" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.351764 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-etc-openvswitch\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.351796 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d533df4c-38b2-42b2-9cb2-b82afac33795-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zkvrz\" (UID: \"d533df4c-38b2-42b2-9cb2-b82afac33795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.351855 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79cqw\" (UniqueName: \"kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-kube-api-access-79cqw\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.351913 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-var-lib-openvswitch\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.351910 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-ovnkube-script-lib\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.351946 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-etc-systemd\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.352008 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6kdm\" (UniqueName: \"kubernetes.io/projected/a6466fc4-ed3d-4031-ba11-5606bd1af74b-kube-api-access-s6kdm\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.352017 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-trusted-ca\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.352044 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrn2g\" (UniqueName: \"kubernetes.io/projected/ea7bdafa-8d46-4b28-b93b-b73352b2e269-kube-api-access-mrn2g\") pod \"cluster-samples-operator-6ddb6c8b94-q79vl\" (UID: \"ea7bdafa-8d46-4b28-b93b-b73352b2e269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q79vl" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.352149 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bf085d57-4ae4-4add-9402-9ee1ce5d795a-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-slvhl\" (UID: \"bf085d57-4ae4-4add-9402-9ee1ce5d795a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-slvhl" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.352223 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-node-log\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.352305 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bf085d57-4ae4-4add-9402-9ee1ce5d795a-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-slvhl\" (UID: \"bf085d57-4ae4-4add-9402-9ee1ce5d795a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-slvhl" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.352329 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-lib-modules\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.352381 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sm2jg\" (UID: \"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee\") " pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.352415 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvrrb\" (UniqueName: \"kubernetes.io/projected/32d5aaf4-3430-4e5b-aca5-f0d720ce8657-kube-api-access-nvrrb\") pod \"service-ca-5fffc97bdf-9vrkp\" (UID: \"32d5aaf4-3430-4e5b-aca5-f0d720ce8657\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9vrkp" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.352495 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8d74c5e4-f631-4eae-84d5-84749b53ccf8-config\") pod \"service-ca-operator-544d749c8c-nvj5v\" (UID: \"8d74c5e4-f631-4eae-84d5-84749b53ccf8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nvj5v" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.352525 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fbfeab8a-b74e-4188-b565-b56442a01b79-registry-certificates\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.352552 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-registry-tls\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.352578 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-host-cni-bin\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.352603 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e637e650-5fc6-4adb-a97a-47d924c30f97-hosts-file\") pod \"node-resolver-q4gsb\" (UID: \"e637e650-5fc6-4adb-a97a-47d924c30f97\") " pod="openshift-dns/node-resolver-q4gsb" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.352637 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6f219713-6fd8-4580-b1c0-24e2a929c258-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zgd4k\" (UID: \"6f219713-6fd8-4580-b1c0-24e2a929c258\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zgd4k" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.352660 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fbfeab8a-b74e-4188-b565-b56442a01b79-registry-certificates\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.352679 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-host-var-lib-cni-multus\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.352695 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8d74c5e4-f631-4eae-84d5-84749b53ccf8-config\") pod \"service-ca-operator-544d749c8c-nvj5v\" (UID: \"8d74c5e4-f631-4eae-84d5-84749b53ccf8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nvj5v" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.352705 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61060804-c519-490a-ac90-674792d50174-metrics-tls\") pod \"dns-default-2bk97\" (UID: \"61060804-c519-490a-ac90-674792d50174\") " pod="openshift-dns/dns-default-2bk97" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.352738 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e9dee2ef-6437-4963-8e27-da5fb9c75928-agent-certs\") pod \"konnectivity-agent-dj8cg\" (UID: \"e9dee2ef-6437-4963-8e27-da5fb9c75928\") " pod="kube-system/konnectivity-agent-dj8cg" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.352789 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d533df4c-38b2-42b2-9cb2-b82afac33795-device-dir\") pod \"aws-ebs-csi-driver-node-zkvrz\" (UID: \"d533df4c-38b2-42b2-9cb2-b82afac33795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.352816 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd-cni-binary-copy\") pod \"multus-additional-cni-plugins-kfkzp\" (UID: \"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd\") " pod="openshift-multus/multus-additional-cni-plugins-kfkzp" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.352855 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-495nb\" (UniqueName: \"kubernetes.io/projected/307e48f6-4a66-4c9c-a4ff-848cce9d3e0e-kube-api-access-495nb\") pod \"console-operator-7cd56b8dbc-5xzwg\" (UID: \"307e48f6-4a66-4c9c-a4ff-848cce9d3e0e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.352886 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-host-run-netns\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.352921 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e637e650-5fc6-4adb-a97a-47d924c30f97-tmp-dir\") pod \"node-resolver-q4gsb\" (UID: \"e637e650-5fc6-4adb-a97a-47d924c30f97\") " pod="openshift-dns/node-resolver-q4gsb" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.352950 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-crio-socket\") pod \"insights-runtime-extractor-sm2jg\" (UID: \"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee\") " pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.352977 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ef43449c-b7dc-4e84-b785-63201f80d4da-dbus\") pod \"global-pull-secret-syncer-2xrr4\" (UID: \"ef43449c-b7dc-4e84-b785-63201f80d4da\") " pod="kube-system/global-pull-secret-syncer-2xrr4" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.353020 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-host-kubelet\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.353049 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e9dee2ef-6437-4963-8e27-da5fb9c75928-agent-certs\") pod \"konnectivity-agent-dj8cg\" (UID: \"e9dee2ef-6437-4963-8e27-da5fb9c75928\") " pod="kube-system/konnectivity-agent-dj8cg" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.353054 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e637e650-5fc6-4adb-a97a-47d924c30f97-tmp-dir\") pod \"node-resolver-q4gsb\" (UID: \"e637e650-5fc6-4adb-a97a-47d924c30f97\") " pod="openshift-dns/node-resolver-q4gsb" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.353087 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6m64\" (UniqueName: \"kubernetes.io/projected/dd3829b5-04dc-4112-b2d9-087d136ba9ec-kube-api-access-f6m64\") pod \"insights-operator-7c4986d449-j66qt\" (UID: \"dd3829b5-04dc-4112-b2d9-087d136ba9ec\") " pod="openshift-insights/insights-operator-7c4986d449-j66qt" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.353096 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd-cni-binary-copy\") pod \"multus-additional-cni-plugins-kfkzp\" (UID: \"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd\") " pod="openshift-multus/multus-additional-cni-plugins-kfkzp" Nov 03 14:11:07.353414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.353105 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl2f4\" (UniqueName: \"kubernetes.io/projected/3fe332c5-8991-4d94-9b67-c9017f4945b8-kube-api-access-xl2f4\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.368035 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.368008 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmwwz\" (UniqueName: \"kubernetes.io/projected/396e7be7-0e72-4c32-a04a-22bd9bd708e1-kube-api-access-fmwwz\") pod \"iptables-alerter-65nrx\" (UID: \"396e7be7-0e72-4c32-a04a-22bd9bd708e1\") " pod="openshift-network-operator/iptables-alerter-65nrx" Nov 03 14:11:07.368166 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.368105 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdggk\" (UniqueName: \"kubernetes.io/projected/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-kube-api-access-hdggk\") pod \"insights-runtime-extractor-sm2jg\" (UID: \"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee\") " pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:07.368631 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.368609 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr7zc\" (UniqueName: \"kubernetes.io/projected/e637e650-5fc6-4adb-a97a-47d924c30f97-kube-api-access-mr7zc\") pod \"node-resolver-q4gsb\" (UID: \"e637e650-5fc6-4adb-a97a-47d924c30f97\") " pod="openshift-dns/node-resolver-q4gsb" Nov 03 14:11:07.369040 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.369020 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5qqj\" (UniqueName: \"kubernetes.io/projected/60d2d1a6-a89f-4a9a-acb9-e85ce2c8f6ce-kube-api-access-w5qqj\") pod \"migrator-66df5d496b-pn8kf\" (UID: \"60d2d1a6-a89f-4a9a-acb9-e85ce2c8f6ce\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pn8kf" Nov 03 14:11:07.369198 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.369175 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwf5m\" (UniqueName: \"kubernetes.io/projected/8d74c5e4-f631-4eae-84d5-84749b53ccf8-kube-api-access-nwf5m\") pod \"service-ca-operator-544d749c8c-nvj5v\" (UID: \"8d74c5e4-f631-4eae-84d5-84749b53ccf8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nvj5v" Nov 03 14:11:07.369198 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.369190 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfv25\" (UniqueName: \"kubernetes.io/projected/d533df4c-38b2-42b2-9cb2-b82afac33795-kube-api-access-sfv25\") pod \"aws-ebs-csi-driver-node-zkvrz\" (UID: \"d533df4c-38b2-42b2-9cb2-b82afac33795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:11:07.369359 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.369182 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-626zm\" (UniqueName: \"kubernetes.io/projected/840ddc67-db99-4d3f-ba2d-51632ac3912c-kube-api-access-626zm\") pod \"network-check-source-6f78d59f99-rjz49\" (UID: \"840ddc67-db99-4d3f-ba2d-51632ac3912c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rjz49" Nov 03 14:11:07.369480 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.369463 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n295\" (UniqueName: \"kubernetes.io/projected/11d16990-ce0c-458f-ab1e-e9a3fed3221b-kube-api-access-8n295\") pod \"volume-data-source-validator-58c9f5f46d-v9mxv\" (UID: \"11d16990-ce0c-458f-ab1e-e9a3fed3221b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-v9mxv" Nov 03 14:11:07.369687 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.369669 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkjrd\" (UniqueName: \"kubernetes.io/projected/328dde79-0c85-47a3-b3f8-38d8cc8614c0-kube-api-access-dkjrd\") pod \"network-check-target-xkhr9\" (UID: \"328dde79-0c85-47a3-b3f8-38d8cc8614c0\") " pod="openshift-network-diagnostics/network-check-target-xkhr9" Nov 03 14:11:07.369772 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.369756 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrqb5\" (UniqueName: \"kubernetes.io/projected/63bfb9d0-9e87-44c6-a6f7-2f7b992e20f6-kube-api-access-hrqb5\") pod \"ingress-canary-qnjdm\" (UID: \"63bfb9d0-9e87-44c6-a6f7-2f7b992e20f6\") " pod="openshift-ingress-canary/ingress-canary-qnjdm" Nov 03 14:11:07.369828 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.369788 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5m26\" (UniqueName: \"kubernetes.io/projected/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d-kube-api-access-m5m26\") pod \"periodic-gathering-fzk44-mlg6z\" (UID: \"7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d\") " pod="openshift-insights/periodic-gathering-fzk44-mlg6z" Nov 03 14:11:07.370248 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.370206 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkrl8\" (UniqueName: \"kubernetes.io/projected/f52cae01-7951-4d9e-9a57-9ba5bcdf6857-kube-api-access-fkrl8\") pod \"network-metrics-daemon-tc5nw\" (UID: \"f52cae01-7951-4d9e-9a57-9ba5bcdf6857\") " pod="openshift-multus/network-metrics-daemon-tc5nw" Nov 03 14:11:07.370479 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.370458 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd5tr\" (UniqueName: \"kubernetes.io/projected/47c859b2-a25b-45e5-acdc-0632d472c097-kube-api-access-pd5tr\") pod \"kas-connection-checker-685895c7c9-pkhg9\" (UID: \"47c859b2-a25b-45e5-acdc-0632d472c097\") " pod="kube-system/kas-connection-checker-685895c7c9-pkhg9" Nov 03 14:11:07.370547 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.370536 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-bound-sa-token\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:11:07.370711 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.370690 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr24x\" (UniqueName: \"kubernetes.io/projected/1da80d21-c62b-44ec-a3fe-64633b5de401-kube-api-access-nr24x\") pod \"migrator-66df5d496b-c8hlq\" (UID: \"1da80d21-c62b-44ec-a3fe-64633b5de401\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-c8hlq" Nov 03 14:11:07.371186 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.371153 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-bound-sa-token\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:11:07.371296 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.371260 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtr9w\" (UniqueName: \"kubernetes.io/projected/bdf183c4-a9ee-4ea8-ace1-50f307114328-kube-api-access-vtr9w\") pod \"cluster-monitoring-operator-884576c57-v6pjk\" (UID: \"bdf183c4-a9ee-4ea8-ace1-50f307114328\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v6pjk" Nov 03 14:11:07.371296 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.371276 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k4jz\" (UniqueName: \"kubernetes.io/projected/36065516-87ab-4057-b198-439305606d12-kube-api-access-8k4jz\") pod \"router-default-6c7f54c9b7-t6wbf\" (UID: \"36065516-87ab-4057-b198-439305606d12\") " pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:11:07.371428 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.371317 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbw2q\" (UniqueName: \"kubernetes.io/projected/bf085d57-4ae4-4add-9402-9ee1ce5d795a-kube-api-access-pbw2q\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-slvhl\" (UID: \"bf085d57-4ae4-4add-9402-9ee1ce5d795a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-slvhl" Nov 03 14:11:07.371428 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.371320 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4w2t\" (UniqueName: \"kubernetes.io/projected/61060804-c519-490a-ac90-674792d50174-kube-api-access-h4w2t\") pod \"dns-default-2bk97\" (UID: \"61060804-c519-490a-ac90-674792d50174\") " pod="openshift-dns/dns-default-2bk97" Nov 03 14:11:07.371428 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.371368 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vktv4\" (UniqueName: \"kubernetes.io/projected/4d66aab2-4671-4bbd-a4cc-b9dd9c5a766d-kube-api-access-vktv4\") pod \"kas-connection-checker-685895c7c9-wcb9f\" (UID: \"4d66aab2-4671-4bbd-a4cc-b9dd9c5a766d\") " pod="kube-system/kas-connection-checker-685895c7c9-wcb9f" Nov 03 14:11:07.371575 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.371503 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbt8t\" (UniqueName: \"kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-kube-api-access-fbt8t\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:11:07.371754 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.371733 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgm54\" (UniqueName: \"kubernetes.io/projected/e68be0c4-d4ee-4da5-82e3-82372f811841-kube-api-access-wgm54\") pod \"node-ca-t4b94\" (UID: \"e68be0c4-d4ee-4da5-82e3-82372f811841\") " pod="openshift-image-registry/node-ca-t4b94" Nov 03 14:11:07.371790 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.371774 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7782\" (UniqueName: \"kubernetes.io/projected/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-kube-api-access-g7782\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.371872 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.371857 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp5h5\" (UniqueName: \"kubernetes.io/projected/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd-kube-api-access-vp5h5\") pod \"multus-additional-cni-plugins-kfkzp\" (UID: \"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd\") " pod="openshift-multus/multus-additional-cni-plugins-kfkzp" Nov 03 14:11:07.372500 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.372449 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pp5r\" (UniqueName: \"kubernetes.io/projected/00a412a1-3764-404b-9e6d-3a2bbb200781-kube-api-access-9pp5r\") pod \"kas-connection-checker-685895c7c9-g28dw\" (UID: \"00a412a1-3764-404b-9e6d-3a2bbb200781\") " pod="kube-system/kas-connection-checker-685895c7c9-g28dw" Nov 03 14:11:07.372936 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.372916 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl2f4\" (UniqueName: \"kubernetes.io/projected/3fe332c5-8991-4d94-9b67-c9017f4945b8-kube-api-access-xl2f4\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.373114 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.373099 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvrrb\" (UniqueName: \"kubernetes.io/projected/32d5aaf4-3430-4e5b-aca5-f0d720ce8657-kube-api-access-nvrrb\") pod \"service-ca-5fffc97bdf-9vrkp\" (UID: \"32d5aaf4-3430-4e5b-aca5-f0d720ce8657\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9vrkp" Nov 03 14:11:07.377260 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.374001 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6m64\" (UniqueName: \"kubernetes.io/projected/dd3829b5-04dc-4112-b2d9-087d136ba9ec-kube-api-access-f6m64\") pod \"insights-operator-7c4986d449-j66qt\" (UID: \"dd3829b5-04dc-4112-b2d9-087d136ba9ec\") " pod="openshift-insights/insights-operator-7c4986d449-j66qt" Nov 03 14:11:07.377260 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.374345 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6kdm\" (UniqueName: \"kubernetes.io/projected/a6466fc4-ed3d-4031-ba11-5606bd1af74b-kube-api-access-s6kdm\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:11:07.377260 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.374374 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-495nb\" (UniqueName: \"kubernetes.io/projected/307e48f6-4a66-4c9c-a4ff-848cce9d3e0e-kube-api-access-495nb\") pod \"console-operator-7cd56b8dbc-5xzwg\" (UID: \"307e48f6-4a66-4c9c-a4ff-848cce9d3e0e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" Nov 03 14:11:07.377260 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.374684 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrn2g\" (UniqueName: \"kubernetes.io/projected/ea7bdafa-8d46-4b28-b93b-b73352b2e269-kube-api-access-mrn2g\") pod \"cluster-samples-operator-6ddb6c8b94-q79vl\" (UID: \"ea7bdafa-8d46-4b28-b93b-b73352b2e269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q79vl" Nov 03 14:11:07.377260 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.374887 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79cqw\" (UniqueName: \"kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-kube-api-access-79cqw\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:11:07.452661 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.452582 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-slvhl_bf085d57-4ae4-4add-9402-9ee1ce5d795a/kube-storage-version-migrator-operator/0.log" Nov 03 14:11:07.454120 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454101 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd-system-cni-dir\") pod \"multus-additional-cni-plugins-kfkzp\" (UID: \"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd\") " pod="openshift-multus/multus-additional-cni-plugins-kfkzp" Nov 03 14:11:07.454190 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454129 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kfkzp\" (UID: \"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd\") " pod="openshift-multus/multus-additional-cni-plugins-kfkzp" Nov 03 14:11:07.454190 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454144 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-sys\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:11:07.454190 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454159 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-etc-modprobe-d\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:11:07.454190 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454179 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-multus-socket-dir-parent\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.454356 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454201 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-host-run-k8s-cni-cncf-io\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.454356 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454224 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-etc-kubernetes\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.454356 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454226 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd-system-cni-dir\") pod \"multus-additional-cni-plugins-kfkzp\" (UID: \"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd\") " pod="openshift-multus/multus-additional-cni-plugins-kfkzp" Nov 03 14:11:07.454356 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454247 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-etc-kubernetes\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:11:07.454356 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454245 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-sys\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:11:07.454356 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454245 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kfkzp\" (UID: \"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd\") " pod="openshift-multus/multus-additional-cni-plugins-kfkzp" Nov 03 14:11:07.454356 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454298 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-etc-kubernetes\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.454356 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454313 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-multus-socket-dir-parent\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.454356 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454329 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-host-run-k8s-cni-cncf-io\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.454356 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454313 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-etc-kubernetes\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:11:07.454356 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454349 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-etc-modprobe-d\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:11:07.454768 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454360 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-run\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:11:07.454768 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454389 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-run\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:11:07.454768 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454409 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-host-run-netns\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.454768 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454423 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-hostroot\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.454768 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454438 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ea7bdafa-8d46-4b28-b93b-b73352b2e269-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-q79vl\" (UID: \"ea7bdafa-8d46-4b28-b93b-b73352b2e269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q79vl" Nov 03 14:11:07.454768 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454458 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-hostroot\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.454768 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454459 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-multus-conf-dir\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.454768 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454477 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-multus-conf-dir\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.454768 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454439 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-host-run-netns\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.454768 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454491 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.454768 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454517 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d533df4c-38b2-42b2-9cb2-b82afac33795-registration-dir\") pod \"aws-ebs-csi-driver-node-zkvrz\" (UID: \"d533df4c-38b2-42b2-9cb2-b82afac33795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:11:07.454768 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454522 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.454768 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454532 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-host-cni-netd\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.454768 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454549 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-etc-sysctl-conf\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:11:07.454768 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454567 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-var-lib-kubelet\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:11:07.454768 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454576 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d533df4c-38b2-42b2-9cb2-b82afac33795-registration-dir\") pod \"aws-ebs-csi-driver-node-zkvrz\" (UID: \"d533df4c-38b2-42b2-9cb2-b82afac33795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:11:07.454768 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454583 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sm2jg\" (UID: \"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee\") " pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:07.454768 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454590 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-host-cni-netd\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.454768 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454628 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-var-lib-kubelet\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:11:07.454768 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454666 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-cnibin\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.454768 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:07.454671 5060 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 14:11:07.454768 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454697 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd-cnibin\") pod \"multus-additional-cni-plugins-kfkzp\" (UID: \"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd\") " pod="openshift-multus/multus-additional-cni-plugins-kfkzp" Nov 03 14:11:07.454768 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454732 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36065516-87ab-4057-b198-439305606d12-service-ca-bundle\") pod \"router-default-6c7f54c9b7-t6wbf\" (UID: \"36065516-87ab-4057-b198-439305606d12\") " pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:11:07.454768 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454697 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-etc-sysctl-conf\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:11:07.454768 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454743 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-cnibin\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.454768 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:07.454757 5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-insights-runtime-extractor-tls podName:3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee nodeName:}" failed. No retries permitted until 2026-11-03 14:11:07.954737319 +0000 UTC m=+1.843526902 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sm2jg" (UID: "3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee") : secret "insights-runtime-extractor-tls" not found Nov 03 14:11:07.454768 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454758 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd-cnibin\") pod \"multus-additional-cni-plugins-kfkzp\" (UID: \"1f5e798e-5caa-4d8f-a1d3-bb41a4efa4dd\") " pod="openshift-multus/multus-additional-cni-plugins-kfkzp" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454794 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bdf183c4-a9ee-4ea8-ace1-50f307114328-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-v6pjk\" (UID: \"bdf183c4-a9ee-4ea8-ace1-50f307114328\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v6pjk" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454859 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-registry-tls\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:07.454871 5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/36065516-87ab-4057-b198-439305606d12-service-ca-bundle podName:36065516-87ab-4057-b198-439305606d12 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:07.954854324 +0000 UTC m=+1.843643896 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/36065516-87ab-4057-b198-439305606d12-service-ca-bundle") pod "router-default-6c7f54c9b7-t6wbf" (UID: "36065516-87ab-4057-b198-439305606d12") : configmap references non-existent config key: service-ca.crt Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:07.454892 5060 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454906 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d533df4c-38b2-42b2-9cb2-b82afac33795-etc-selinux\") pod \"aws-ebs-csi-driver-node-zkvrz\" (UID: \"d533df4c-38b2-42b2-9cb2-b82afac33795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:07.454921 5060 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454929 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d533df4c-38b2-42b2-9cb2-b82afac33795-sys-fs\") pod \"aws-ebs-csi-driver-node-zkvrz\" (UID: \"d533df4c-38b2-42b2-9cb2-b82afac33795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:07.454933 5060 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d7c596f48-qrcsg: secret "image-registry-tls" not found Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:07.454948 5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdf183c4-a9ee-4ea8-ace1-50f307114328-cluster-monitoring-operator-tls podName:bdf183c4-a9ee-4ea8-ace1-50f307114328 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:07.954933808 +0000 UTC m=+1.843723378 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bdf183c4-a9ee-4ea8-ace1-50f307114328-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-v6pjk" (UID: "bdf183c4-a9ee-4ea8-ace1-50f307114328") : secret "cluster-monitoring-operator-tls" not found Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:07.454982 5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-registry-tls podName:b3260ef2-8ff4-4b70-a3a0-e5386a27d210 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:07.954967246 +0000 UTC m=+1.843756819 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-registry-tls") pod "image-registry-6d7c596f48-qrcsg" (UID: "b3260ef2-8ff4-4b70-a3a0-e5386a27d210") : secret "image-registry-tls" not found Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454973 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d533df4c-38b2-42b2-9cb2-b82afac33795-etc-selinux\") pod \"aws-ebs-csi-driver-node-zkvrz\" (UID: \"d533df4c-38b2-42b2-9cb2-b82afac33795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.454988 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-host-run-ovn-kubernetes\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455009 5060 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455020 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-host-run-ovn-kubernetes\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455029 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d533df4c-38b2-42b2-9cb2-b82afac33795-sys-fs\") pod \"aws-ebs-csi-driver-node-zkvrz\" (UID: \"d533df4c-38b2-42b2-9cb2-b82afac33795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455041 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-system-cni-dir\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455069 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ef43449c-b7dc-4e84-b785-63201f80d4da-kubelet-config\") pod \"global-pull-secret-syncer-2xrr4\" (UID: \"ef43449c-b7dc-4e84-b785-63201f80d4da\") " pod="kube-system/global-pull-secret-syncer-2xrr4" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455094 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-systemd-units\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455119 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-multus-cni-dir\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455141 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-host-var-lib-cni-bin\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455147 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ef43449c-b7dc-4e84-b785-63201f80d4da-kubelet-config\") pod \"global-pull-secret-syncer-2xrr4\" (UID: \"ef43449c-b7dc-4e84-b785-63201f80d4da\") " pod="kube-system/global-pull-secret-syncer-2xrr4" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455163 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-host-var-lib-kubelet\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455141 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-system-cni-dir\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455189 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d533df4c-38b2-42b2-9cb2-b82afac33795-socket-dir\") pod \"aws-ebs-csi-driver-node-zkvrz\" (UID: \"d533df4c-38b2-42b2-9cb2-b82afac33795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455194 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-multus-cni-dir\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455213 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f52cae01-7951-4d9e-9a57-9ba5bcdf6857-metrics-certs\") pod \"network-metrics-daemon-tc5nw\" (UID: \"f52cae01-7951-4d9e-9a57-9ba5bcdf6857\") " pod="openshift-multus/network-metrics-daemon-tc5nw" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455221 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-host-var-lib-cni-bin\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455230 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-host-var-lib-kubelet\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455190 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-systemd-units\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455248 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d533df4c-38b2-42b2-9cb2-b82afac33795-socket-dir\") pod \"aws-ebs-csi-driver-node-zkvrz\" (UID: \"d533df4c-38b2-42b2-9cb2-b82afac33795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455253 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-etc-sysconfig\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455276 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-etc-sysctl-d\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455295 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e68be0c4-d4ee-4da5-82e3-82372f811841-host\") pod \"node-ca-t4b94\" (UID: \"e68be0c4-d4ee-4da5-82e3-82372f811841\") " pod="openshift-image-registry/node-ca-t4b94" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455310 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/396e7be7-0e72-4c32-a04a-22bd9bd708e1-host-slash\") pod \"iptables-alerter-65nrx\" (UID: \"396e7be7-0e72-4c32-a04a-22bd9bd708e1\") " pod="openshift-network-operator/iptables-alerter-65nrx" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455322 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-etc-sysconfig\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455332 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-run-systemd\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455366 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-run-systemd\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455391 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-run-openvswitch\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455404 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/396e7be7-0e72-4c32-a04a-22bd9bd708e1-host-slash\") pod \"iptables-alerter-65nrx\" (UID: \"396e7be7-0e72-4c32-a04a-22bd9bd708e1\") " pod="openshift-network-operator/iptables-alerter-65nrx" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455361 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e68be0c4-d4ee-4da5-82e3-82372f811841-host\") pod \"node-ca-t4b94\" (UID: \"e68be0c4-d4ee-4da5-82e3-82372f811841\") " pod="openshift-image-registry/node-ca-t4b94" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455416 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-run-ovn\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455440 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-host-run-multus-certs\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455450 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-run-ovn\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455466 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-log-socket\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455470 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-etc-sysctl-d\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455486 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-run-openvswitch\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455492 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36065516-87ab-4057-b198-439305606d12-metrics-certs\") pod \"router-default-6c7f54c9b7-t6wbf\" (UID: \"36065516-87ab-4057-b198-439305606d12\") " pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455511 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-host-run-multus-certs\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455520 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-host-slash\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455525 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-log-socket\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455543 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-host\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455568 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-host-slash\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455569 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-etc-openvswitch\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455595 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d533df4c-38b2-42b2-9cb2-b82afac33795-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zkvrz\" (UID: \"d533df4c-38b2-42b2-9cb2-b82afac33795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455617 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-host\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:07.455597 5060 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455629 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-etc-openvswitch\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455635 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-var-lib-openvswitch\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:07.455666 5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36065516-87ab-4057-b198-439305606d12-metrics-certs podName:36065516-87ab-4057-b198-439305606d12 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:07.955651616 +0000 UTC m=+1.844441175 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/36065516-87ab-4057-b198-439305606d12-metrics-certs") pod "router-default-6c7f54c9b7-t6wbf" (UID: "36065516-87ab-4057-b198-439305606d12") : secret "router-metrics-certs-default" not found Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455671 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-var-lib-openvswitch\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455684 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-etc-systemd\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455668 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d533df4c-38b2-42b2-9cb2-b82afac33795-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zkvrz\" (UID: \"d533df4c-38b2-42b2-9cb2-b82afac33795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455713 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-node-log\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455750 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-node-log\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455766 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-lib-modules\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455795 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sm2jg\" (UID: \"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee\") " pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455771 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-etc-systemd\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455820 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-registry-tls\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455874 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-host-cni-bin\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455897 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e637e650-5fc6-4adb-a97a-47d924c30f97-hosts-file\") pod \"node-resolver-q4gsb\" (UID: \"e637e650-5fc6-4adb-a97a-47d924c30f97\") " pod="openshift-dns/node-resolver-q4gsb" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455899 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a6466fc4-ed3d-4031-ba11-5606bd1af74b-lib-modules\") pod \"tuned-xcplz\" (UID: \"a6466fc4-ed3d-4031-ba11-5606bd1af74b\") " pod="openshift-cluster-node-tuning-operator/tuned-xcplz" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:07.455913 5060 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455921 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6f219713-6fd8-4580-b1c0-24e2a929c258-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zgd4k\" (UID: \"6f219713-6fd8-4580-b1c0-24e2a929c258\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zgd4k" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455947 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-host-var-lib-cni-multus\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:07.455928 5060 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57647df665-9q52v: secret "image-registry-tls" not found Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455975 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61060804-c519-490a-ac90-674792d50174-metrics-tls\") pod \"dns-default-2bk97\" (UID: \"61060804-c519-490a-ac90-674792d50174\") " pod="openshift-dns/dns-default-2bk97" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:07.456019 5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-registry-tls podName:fbfeab8a-b74e-4188-b565-b56442a01b79 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:07.956004578 +0000 UTC m=+1.844794152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-registry-tls") pod "image-registry-57647df665-9q52v" (UID: "fbfeab8a-b74e-4188-b565-b56442a01b79") : secret "image-registry-tls" not found Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.456051 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d533df4c-38b2-42b2-9cb2-b82afac33795-device-dir\") pod \"aws-ebs-csi-driver-node-zkvrz\" (UID: \"d533df4c-38b2-42b2-9cb2-b82afac33795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:07.456069 5060 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.456079 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-host-run-netns\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.456104 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-crio-socket\") pod \"insights-runtime-extractor-sm2jg\" (UID: \"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee\") " pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:07.456116 5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61060804-c519-490a-ac90-674792d50174-metrics-tls podName:61060804-c519-490a-ac90-674792d50174 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:07.956104134 +0000 UTC m=+1.844893725 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61060804-c519-490a-ac90-674792d50174-metrics-tls") pod "dns-default-2bk97" (UID: "61060804-c519-490a-ac90-674792d50174") : secret "dns-default-metrics-tls" not found Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.456139 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ef43449c-b7dc-4e84-b785-63201f80d4da-dbus\") pod \"global-pull-secret-syncer-2xrr4\" (UID: \"ef43449c-b7dc-4e84-b785-63201f80d4da\") " pod="kube-system/global-pull-secret-syncer-2xrr4" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.456148 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-crio-socket\") pod \"insights-runtime-extractor-sm2jg\" (UID: \"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee\") " pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.456160 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-host-kubelet\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:07.455956 5060 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:07.456246 5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-kube-rbac-proxy-cm podName:3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee nodeName:}" failed. No retries permitted until 2026-11-03 14:11:07.95623609 +0000 UTC m=+1.845025689 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sm2jg" (UID: "3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee") : configmap "kube-rbac-proxy" not found Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.456242 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-host-run-netns\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.456243 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e637e650-5fc6-4adb-a97a-47d924c30f97-hosts-file\") pod \"node-resolver-q4gsb\" (UID: \"e637e650-5fc6-4adb-a97a-47d924c30f97\") " pod="openshift-dns/node-resolver-q4gsb" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.456288 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ef43449c-b7dc-4e84-b785-63201f80d4da-dbus\") pod \"global-pull-secret-syncer-2xrr4\" (UID: \"ef43449c-b7dc-4e84-b785-63201f80d4da\") " pod="kube-system/global-pull-secret-syncer-2xrr4" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.456204 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d533df4c-38b2-42b2-9cb2-b82afac33795-device-dir\") pod \"aws-ebs-csi-driver-node-zkvrz\" (UID: \"d533df4c-38b2-42b2-9cb2-b82afac33795\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkvrz" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.455954 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-host-cni-bin\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.456293 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce-host-kubelet\") pod \"ovnkube-node-w6d6z\" (UID: \"71c2e12f-32f9-4ab1-b74c-bc1dd28d4cce\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:07.456370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.456342 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3fe332c5-8991-4d94-9b67-c9017f4945b8-host-var-lib-cni-multus\") pod \"multus-q5cqm\" (UID: \"3fe332c5-8991-4d94-9b67-c9017f4945b8\") " pod="openshift-multus/multus-q5cqm" Nov 03 14:11:07.459254 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.458132 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f52cae01-7951-4d9e-9a57-9ba5bcdf6857-metrics-certs\") pod \"network-metrics-daemon-tc5nw\" (UID: \"f52cae01-7951-4d9e-9a57-9ba5bcdf6857\") " pod="openshift-multus/network-metrics-daemon-tc5nw" Nov 03 14:11:07.459254 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.458301 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6f219713-6fd8-4580-b1c0-24e2a929c258-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zgd4k\" (UID: \"6f219713-6fd8-4580-b1c0-24e2a929c258\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zgd4k" Nov 03 14:11:07.459254 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.458491 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ea7bdafa-8d46-4b28-b93b-b73352b2e269-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-q79vl\" (UID: \"ea7bdafa-8d46-4b28-b93b-b73352b2e269\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q79vl" Nov 03 14:11:07.550012 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.549984 5060 scope.go:122] "RemoveContainer" containerID="d87a44a09d0ad13a61108d4537be53ab5164f5519ade849da1c1ba1e459d0490" Nov 03 14:11:07.553327 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.553309 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q79vl" Nov 03 14:11:07.554398 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.554383 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-zgd4k" Nov 03 14:11:07.558901 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.558883 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc5nw" Nov 03 14:11:07.893041 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.893003 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q79vl"] Nov 03 14:11:07.938559 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.938530 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-zgd4k"] Nov 03 14:11:07.942073 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:07.942039 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f219713_6fd8_4580_b1c0_24e2a929c258.slice/crio-85a4f30fe139e74934e797e6110f1595d82245f09ba5e947c7e8d0438fd92992 WatchSource:0}: Error finding container 85a4f30fe139e74934e797e6110f1595d82245f09ba5e947c7e8d0438fd92992: Status 404 returned error can't find the container with id 85a4f30fe139e74934e797e6110f1595d82245f09ba5e947c7e8d0438fd92992 Nov 03 14:11:07.944128 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.944112 5060 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 14:11:07.960909 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.960882 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36065516-87ab-4057-b198-439305606d12-service-ca-bundle\") pod \"router-default-6c7f54c9b7-t6wbf\" (UID: \"36065516-87ab-4057-b198-439305606d12\") " pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:11:07.961009 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.960921 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bdf183c4-a9ee-4ea8-ace1-50f307114328-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-v6pjk\" (UID: \"bdf183c4-a9ee-4ea8-ace1-50f307114328\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v6pjk" Nov 03 14:11:07.961068 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:07.961020 5060 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 14:11:07.961068 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:07.961025 5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/36065516-87ab-4057-b198-439305606d12-service-ca-bundle podName:36065516-87ab-4057-b198-439305606d12 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:08.961003844 +0000 UTC m=+2.849793403 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/36065516-87ab-4057-b198-439305606d12-service-ca-bundle") pod "router-default-6c7f54c9b7-t6wbf" (UID: "36065516-87ab-4057-b198-439305606d12") : configmap references non-existent config key: service-ca.crt Nov 03 14:11:07.961174 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.961083 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-registry-tls\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:11:07.961174 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.961149 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36065516-87ab-4057-b198-439305606d12-metrics-certs\") pod \"router-default-6c7f54c9b7-t6wbf\" (UID: \"36065516-87ab-4057-b198-439305606d12\") " pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:11:07.961254 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:07.961169 5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdf183c4-a9ee-4ea8-ace1-50f307114328-cluster-monitoring-operator-tls podName:bdf183c4-a9ee-4ea8-ace1-50f307114328 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:08.961153599 +0000 UTC m=+2.849943169 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bdf183c4-a9ee-4ea8-ace1-50f307114328-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-v6pjk" (UID: "bdf183c4-a9ee-4ea8-ace1-50f307114328") : secret "cluster-monitoring-operator-tls" not found Nov 03 14:11:07.961254 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:07.961218 5060 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 14:11:07.961254 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.961237 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sm2jg\" (UID: \"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee\") " pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:07.961347 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:07.961255 5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36065516-87ab-4057-b198-439305606d12-metrics-certs podName:36065516-87ab-4057-b198-439305606d12 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:08.961244438 +0000 UTC m=+2.850033999 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/36065516-87ab-4057-b198-439305606d12-metrics-certs") pod "router-default-6c7f54c9b7-t6wbf" (UID: "36065516-87ab-4057-b198-439305606d12") : secret "router-metrics-certs-default" not found Nov 03 14:11:07.961347 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:07.961285 5060 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 14:11:07.961347 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:07.961311 5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-kube-rbac-proxy-cm podName:3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee nodeName:}" failed. No retries permitted until 2026-11-03 14:11:08.961304979 +0000 UTC m=+2.850094551 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sm2jg" (UID: "3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee") : configmap "kube-rbac-proxy" not found Nov 03 14:11:07.961347 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.961328 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-registry-tls\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:11:07.961510 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.961353 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61060804-c519-490a-ac90-674792d50174-metrics-tls\") pod \"dns-default-2bk97\" (UID: \"61060804-c519-490a-ac90-674792d50174\") " pod="openshift-dns/dns-default-2bk97" Nov 03 14:11:07.961510 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.961415 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sm2jg\" (UID: \"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee\") " pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:07.961510 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:07.961485 5060 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 14:11:07.961510 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:07.961498 5060 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 14:11:07.961646 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:07.961528 5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61060804-c519-490a-ac90-674792d50174-metrics-tls podName:61060804-c519-490a-ac90-674792d50174 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:08.961513485 +0000 UTC m=+2.850303058 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61060804-c519-490a-ac90-674792d50174-metrics-tls") pod "dns-default-2bk97" (UID: "61060804-c519-490a-ac90-674792d50174") : secret "dns-default-metrics-tls" not found Nov 03 14:11:07.961646 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:07.961547 5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-insights-runtime-extractor-tls podName:3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee nodeName:}" failed. No retries permitted until 2026-11-03 14:11:08.961537207 +0000 UTC m=+2.850326772 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sm2jg" (UID: "3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee") : secret "insights-runtime-extractor-tls" not found Nov 03 14:11:07.963371 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.963349 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tc5nw"] Nov 03 14:11:07.963486 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.963467 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-registry-tls\") pod \"image-registry-6d7c596f48-qrcsg\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:11:07.964168 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:07.963728 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-registry-tls\") pod \"image-registry-57647df665-9q52v\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:11:07.967226 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:07.967202 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf52cae01_7951_4d9e_9a57_9ba5bcdf6857.slice/crio-23a05d4df544ea49edc819695a7a1599c41b35c4ef069870f4cb1be87749b04c WatchSource:0}: Error finding container 23a05d4df544ea49edc819695a7a1599c41b35c4ef069870f4cb1be87749b04c: Status 404 returned error can't find the container with id 23a05d4df544ea49edc819695a7a1599c41b35c4ef069870f4cb1be87749b04c Nov 03 14:11:08.153901 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:08.153869 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:11:08.163192 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:08.163150 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:11:08.310627 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:08.310597 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q79vl" event={"ID":"ea7bdafa-8d46-4b28-b93b-b73352b2e269","Type":"ContainerStarted","Data":"7fd847dc1b36c8a0c3a9f88d45ee462631bb1f47387f63995025abb811defbb3"} Nov 03 14:11:08.311820 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:08.311794 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-zgd4k" event={"ID":"6f219713-6fd8-4580-b1c0-24e2a929c258","Type":"ContainerStarted","Data":"85a4f30fe139e74934e797e6110f1595d82245f09ba5e947c7e8d0438fd92992"} Nov 03 14:11:08.313027 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:08.313003 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tc5nw" event={"ID":"f52cae01-7951-4d9e-9a57-9ba5bcdf6857","Type":"ContainerStarted","Data":"23a05d4df544ea49edc819695a7a1599c41b35c4ef069870f4cb1be87749b04c"} Nov 03 14:11:08.314524 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:08.314503 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/2.log" Nov 03 14:11:08.314920 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:08.314904 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/1.log" Nov 03 14:11:08.314990 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:08.314936 5060 generic.go:356] "Generic (PLEG): container finished" podID="307e48f6-4a66-4c9c-a4ff-848cce9d3e0e" containerID="ecd927bb274859bd54bc1db7522266968c4926ad29d8376341481d3ad15ddaea" exitCode=255 Nov 03 14:11:08.315038 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:08.315010 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" event={"ID":"307e48f6-4a66-4c9c-a4ff-848cce9d3e0e","Type":"ContainerDied","Data":"ecd927bb274859bd54bc1db7522266968c4926ad29d8376341481d3ad15ddaea"} Nov 03 14:11:08.315085 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:08.315067 5060 scope.go:122] "RemoveContainer" containerID="d87a44a09d0ad13a61108d4537be53ab5164f5519ade849da1c1ba1e459d0490" Nov 03 14:11:08.315597 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:08.315579 5060 scope.go:122] "RemoveContainer" containerID="ecd927bb274859bd54bc1db7522266968c4926ad29d8376341481d3ad15ddaea" Nov 03 14:11:08.315780 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:08.315749 5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5xzwg_openshift-console-operator(307e48f6-4a66-4c9c-a4ff-848cce9d3e0e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" podUID="307e48f6-4a66-4c9c-a4ff-848cce9d3e0e" Nov 03 14:11:08.500559 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:08.500517 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57647df665-9q52v"] Nov 03 14:11:08.505406 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:08.505374 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbfeab8a_b74e_4188_b565_b56442a01b79.slice/crio-95901b99c6d073d798f9cf1410ef4a41174bacb7b7ff504c9ede48711d2ff199 WatchSource:0}: Error finding container 95901b99c6d073d798f9cf1410ef4a41174bacb7b7ff504c9ede48711d2ff199: Status 404 returned error can't find the container with id 95901b99c6d073d798f9cf1410ef4a41174bacb7b7ff504c9ede48711d2ff199 Nov 03 14:11:08.524967 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:08.524424 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d7c596f48-qrcsg"] Nov 03 14:11:08.861390 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:08.861346 5060 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" Nov 03 14:11:08.970268 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:08.970230 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sm2jg\" (UID: \"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee\") " pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:08.970424 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:08.970278 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36065516-87ab-4057-b198-439305606d12-service-ca-bundle\") pod \"router-default-6c7f54c9b7-t6wbf\" (UID: \"36065516-87ab-4057-b198-439305606d12\") " pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:11:08.970424 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:08.970319 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bdf183c4-a9ee-4ea8-ace1-50f307114328-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-v6pjk\" (UID: \"bdf183c4-a9ee-4ea8-ace1-50f307114328\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v6pjk" Nov 03 14:11:08.970424 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:08.970356 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36065516-87ab-4057-b198-439305606d12-metrics-certs\") pod \"router-default-6c7f54c9b7-t6wbf\" (UID: \"36065516-87ab-4057-b198-439305606d12\") " pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:11:08.970424 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:08.970386 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sm2jg\" (UID: \"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee\") " pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:08.970424 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:08.970400 5060 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 14:11:08.970424 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:08.970417 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61060804-c519-490a-ac90-674792d50174-metrics-tls\") pod \"dns-default-2bk97\" (UID: \"61060804-c519-490a-ac90-674792d50174\") " pod="openshift-dns/dns-default-2bk97" Nov 03 14:11:08.970704 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:08.970457 5060 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 14:11:08.970704 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:08.970460 5060 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 14:11:08.970704 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:08.970486 5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-insights-runtime-extractor-tls podName:3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee nodeName:}" failed. No retries permitted until 2026-11-03 14:11:10.970465483 +0000 UTC m=+4.859255057 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sm2jg" (UID: "3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee") : secret "insights-runtime-extractor-tls" not found Nov 03 14:11:08.970704 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:08.970509 5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36065516-87ab-4057-b198-439305606d12-metrics-certs podName:36065516-87ab-4057-b198-439305606d12 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:10.970493938 +0000 UTC m=+4.859283512 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/36065516-87ab-4057-b198-439305606d12-metrics-certs") pod "router-default-6c7f54c9b7-t6wbf" (UID: "36065516-87ab-4057-b198-439305606d12") : secret "router-metrics-certs-default" not found Nov 03 14:11:08.970704 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:08.970526 5060 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 14:11:08.970704 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:08.970528 5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdf183c4-a9ee-4ea8-ace1-50f307114328-cluster-monitoring-operator-tls podName:bdf183c4-a9ee-4ea8-ace1-50f307114328 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:10.970518919 +0000 UTC m=+4.859308477 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bdf183c4-a9ee-4ea8-ace1-50f307114328-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-v6pjk" (UID: "bdf183c4-a9ee-4ea8-ace1-50f307114328") : secret "cluster-monitoring-operator-tls" not found Nov 03 14:11:08.970704 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:08.970546 5060 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 14:11:08.970704 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:08.970558 5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61060804-c519-490a-ac90-674792d50174-metrics-tls podName:61060804-c519-490a-ac90-674792d50174 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:10.970550385 +0000 UTC m=+4.859339963 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61060804-c519-490a-ac90-674792d50174-metrics-tls") pod "dns-default-2bk97" (UID: "61060804-c519-490a-ac90-674792d50174") : secret "dns-default-metrics-tls" not found Nov 03 14:11:08.970704 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:08.970589 5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-kube-rbac-proxy-cm podName:3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee nodeName:}" failed. No retries permitted until 2026-11-03 14:11:10.9705716 +0000 UTC m=+4.859361180 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sm2jg" (UID: "3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee") : configmap "kube-rbac-proxy" not found Nov 03 14:11:08.971106 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:08.971083 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36065516-87ab-4057-b198-439305606d12-service-ca-bundle\") pod \"router-default-6c7f54c9b7-t6wbf\" (UID: \"36065516-87ab-4057-b198-439305606d12\") " pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:11:08.996177 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:08.995458 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dj8cg" Nov 03 14:11:09.322140 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:09.321986 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/2.log" Nov 03 14:11:09.322680 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:09.322374 5060 scope.go:122] "RemoveContainer" containerID="ecd927bb274859bd54bc1db7522266968c4926ad29d8376341481d3ad15ddaea" Nov 03 14:11:09.322680 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:09.322543 5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5xzwg_openshift-console-operator(307e48f6-4a66-4c9c-a4ff-848cce9d3e0e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" podUID="307e48f6-4a66-4c9c-a4ff-848cce9d3e0e" Nov 03 14:11:09.324937 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:09.324907 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" event={"ID":"b3260ef2-8ff4-4b70-a3a0-e5386a27d210","Type":"ContainerStarted","Data":"cbdb6cf03b590c5a2a9ce28e2d7babdf7f2a13983310b1564a976a1e873270c9"} Nov 03 14:11:09.325015 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:09.324962 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" event={"ID":"b3260ef2-8ff4-4b70-a3a0-e5386a27d210","Type":"ContainerStarted","Data":"ecb6d105641db003649bba764f30ed8ec95d1fd3e3e0cca57a76f448cdfd294d"} Nov 03 14:11:09.328488 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:09.327158 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tc5nw" event={"ID":"f52cae01-7951-4d9e-9a57-9ba5bcdf6857","Type":"ContainerStarted","Data":"e7464381a1b1bd2d6afc8cd5ff112a8d489692366ad9765e57b5de1a8d7e5e97"} Nov 03 14:11:09.329698 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:09.329665 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57647df665-9q52v" event={"ID":"fbfeab8a-b74e-4188-b565-b56442a01b79","Type":"ContainerStarted","Data":"af30d755532f37de5e52d15dc1fb476952fd52dc3c1bcec2b2e7c2700d5a4763"} Nov 03 14:11:09.329801 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:09.329700 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57647df665-9q52v" event={"ID":"fbfeab8a-b74e-4188-b565-b56442a01b79","Type":"ContainerStarted","Data":"95901b99c6d073d798f9cf1410ef4a41174bacb7b7ff504c9ede48711d2ff199"} Nov 03 14:11:10.337312 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:10.337270 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q79vl" event={"ID":"ea7bdafa-8d46-4b28-b93b-b73352b2e269","Type":"ContainerStarted","Data":"f6e47e1e90617bc59948bb02d776b395c40b8b7d1618fc502897b3398a491da9"} Nov 03 14:11:10.337734 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:10.337348 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q79vl" event={"ID":"ea7bdafa-8d46-4b28-b93b-b73352b2e269","Type":"ContainerStarted","Data":"4a0ebbc48e497d63306003171e4bcb090c73cac781a61a3741e6445b5e6ff9a9"} Nov 03 14:11:10.338798 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:10.338774 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-zgd4k" event={"ID":"6f219713-6fd8-4580-b1c0-24e2a929c258","Type":"ContainerStarted","Data":"44cf8dee5d63db81faf7c150566fe59629a65484e98452fd1acc92db4d48eb9b"} Nov 03 14:11:10.340124 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:10.340103 5060 generic.go:356] "Generic (PLEG): container finished" podID="dd3829b5-04dc-4112-b2d9-087d136ba9ec" containerID="586cf47171ee95b4f449f8b14c62e58862cde12b2db560a5917d6a54dff60ed1" exitCode=0 Nov 03 14:11:10.340212 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:10.340171 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-j66qt" event={"ID":"dd3829b5-04dc-4112-b2d9-087d136ba9ec","Type":"ContainerDied","Data":"586cf47171ee95b4f449f8b14c62e58862cde12b2db560a5917d6a54dff60ed1"} Nov 03 14:11:10.340511 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:10.340491 5060 scope.go:122] "RemoveContainer" containerID="586cf47171ee95b4f449f8b14c62e58862cde12b2db560a5917d6a54dff60ed1" Nov 03 14:11:10.341888 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:10.341849 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tc5nw" event={"ID":"f52cae01-7951-4d9e-9a57-9ba5bcdf6857","Type":"ContainerStarted","Data":"4c46135157f149b0b9208b46f8a12d188e9fcc6eaeb18f82d1c8ad5046e7e1dd"} Nov 03 14:11:10.341888 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:10.341861 5060 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 14:11:10.341888 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:10.341885 5060 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 14:11:10.725263 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:10.725223 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" Nov 03 14:11:10.725752 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:10.725736 5060 scope.go:122] "RemoveContainer" containerID="ecd927bb274859bd54bc1db7522266968c4926ad29d8376341481d3ad15ddaea" Nov 03 14:11:10.726011 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:10.725988 5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5xzwg_openshift-console-operator(307e48f6-4a66-4c9c-a4ff-848cce9d3e0e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" podUID="307e48f6-4a66-4c9c-a4ff-848cce9d3e0e" Nov 03 14:11:10.991365 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:10.991331 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sm2jg\" (UID: \"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee\") " pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:10.991505 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:10.991376 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bdf183c4-a9ee-4ea8-ace1-50f307114328-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-v6pjk\" (UID: \"bdf183c4-a9ee-4ea8-ace1-50f307114328\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v6pjk" Nov 03 14:11:10.991505 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:10.991423 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36065516-87ab-4057-b198-439305606d12-metrics-certs\") pod \"router-default-6c7f54c9b7-t6wbf\" (UID: \"36065516-87ab-4057-b198-439305606d12\") " pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:11:10.991505 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:10.991447 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sm2jg\" (UID: \"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee\") " pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:10.991505 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:10.991473 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61060804-c519-490a-ac90-674792d50174-metrics-tls\") pod \"dns-default-2bk97\" (UID: \"61060804-c519-490a-ac90-674792d50174\") " pod="openshift-dns/dns-default-2bk97" Nov 03 14:11:10.991505 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:10.991477 5060 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 14:11:10.991681 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:10.991559 5060 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 14:11:10.991681 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:10.991572 5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-insights-runtime-extractor-tls podName:3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee nodeName:}" failed. No retries permitted until 2026-11-03 14:11:14.991551572 +0000 UTC m=+8.880341145 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sm2jg" (UID: "3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee") : secret "insights-runtime-extractor-tls" not found Nov 03 14:11:10.991681 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:10.991574 5060 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 14:11:10.991681 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:10.991616 5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61060804-c519-490a-ac90-674792d50174-metrics-tls podName:61060804-c519-490a-ac90-674792d50174 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:14.991603019 +0000 UTC m=+8.880392595 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61060804-c519-490a-ac90-674792d50174-metrics-tls") pod "dns-default-2bk97" (UID: "61060804-c519-490a-ac90-674792d50174") : secret "dns-default-metrics-tls" not found Nov 03 14:11:10.991681 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:10.991633 5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-kube-rbac-proxy-cm podName:3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee nodeName:}" failed. No retries permitted until 2026-11-03 14:11:14.991625688 +0000 UTC m=+8.880415246 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sm2jg" (UID: "3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee") : configmap "kube-rbac-proxy" not found Nov 03 14:11:10.994077 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:10.994049 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bdf183c4-a9ee-4ea8-ace1-50f307114328-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-v6pjk\" (UID: \"bdf183c4-a9ee-4ea8-ace1-50f307114328\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v6pjk" Nov 03 14:11:10.994182 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:10.994049 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36065516-87ab-4057-b198-439305606d12-metrics-certs\") pod \"router-default-6c7f54c9b7-t6wbf\" (UID: \"36065516-87ab-4057-b198-439305606d12\") " pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:11:11.151129 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:11.151088 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:11:11.151271 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:11.151101 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v6pjk" Nov 03 14:11:11.347429 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:11.347344 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-j66qt" event={"ID":"dd3829b5-04dc-4112-b2d9-087d136ba9ec","Type":"ContainerStarted","Data":"e32c8a3ff556b8c5256d83620697a50e2a76d8147164c28802f6b31e7a750786"} Nov 03 14:11:11.470396 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:11.470365 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-v6pjk"] Nov 03 14:11:11.472917 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:11.472893 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6c7f54c9b7-t6wbf"] Nov 03 14:11:11.474385 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:11.474276 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdf183c4_a9ee_4ea8_ace1_50f307114328.slice/crio-3c66cd0b626e724c9d492242dc458b50452866e8df53ecc7b5915b9915772880 WatchSource:0}: Error finding container 3c66cd0b626e724c9d492242dc458b50452866e8df53ecc7b5915b9915772880: Status 404 returned error can't find the container with id 3c66cd0b626e724c9d492242dc458b50452866e8df53ecc7b5915b9915772880 Nov 03 14:11:11.476590 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:11.476564 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36065516_87ab_4057_b198_439305606d12.slice/crio-63181a32b20be1e09208c65fa93477d9b144d4e1c24f6b855e252d60876809eb WatchSource:0}: Error finding container 63181a32b20be1e09208c65fa93477d9b144d4e1c24f6b855e252d60876809eb: Status 404 returned error can't find the container with id 63181a32b20be1e09208c65fa93477d9b144d4e1c24f6b855e252d60876809eb Nov 03 14:11:12.353194 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:12.353162 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v6pjk" event={"ID":"bdf183c4-a9ee-4ea8-ace1-50f307114328","Type":"ContainerStarted","Data":"3c66cd0b626e724c9d492242dc458b50452866e8df53ecc7b5915b9915772880"} Nov 03 14:11:12.355128 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:12.355094 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" event={"ID":"36065516-87ab-4057-b198-439305606d12","Type":"ContainerStarted","Data":"55dba6f2151e6a313cb1c13d85be4f510643b476552693e5910062066af5fb69"} Nov 03 14:11:12.355250 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:12.355144 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" event={"ID":"36065516-87ab-4057-b198-439305606d12","Type":"ContainerStarted","Data":"63181a32b20be1e09208c65fa93477d9b144d4e1c24f6b855e252d60876809eb"} Nov 03 14:11:13.151024 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:13.150992 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:11:13.152109 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:13.152089 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:11:13.154221 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:13.154199 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:11:13.361764 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:13.361683 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v6pjk" event={"ID":"bdf183c4-a9ee-4ea8-ace1-50f307114328","Type":"ContainerStarted","Data":"cdafa836c84ed72232ec838399dbd5573d2caf4b78da3cdb9cbd2a9b29f7818a"} Nov 03 14:11:13.363523 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:13.363499 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6c7f54c9b7-t6wbf" Nov 03 14:11:14.800113 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:14.800082 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:15.029971 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:15.029935 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sm2jg\" (UID: \"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee\") " pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:15.030114 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:15.029977 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61060804-c519-490a-ac90-674792d50174-metrics-tls\") pod \"dns-default-2bk97\" (UID: \"61060804-c519-490a-ac90-674792d50174\") " pod="openshift-dns/dns-default-2bk97" Nov 03 14:11:15.030114 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:15.030031 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sm2jg\" (UID: \"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee\") " pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:15.030454 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:15.030430 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sm2jg\" (UID: \"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee\") " pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:15.032397 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:15.032374 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sm2jg\" (UID: \"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee\") " pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:15.032486 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:15.032403 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61060804-c519-490a-ac90-674792d50174-metrics-tls\") pod \"dns-default-2bk97\" (UID: \"61060804-c519-490a-ac90-674792d50174\") " pod="openshift-dns/dns-default-2bk97" Nov 03 14:11:15.064027 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:15.063970 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2bk97" Nov 03 14:11:15.067108 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:15.067089 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sm2jg" Nov 03 14:11:15.113686 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:15.113640 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:11:15.393245 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:15.393128 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2bk97"] Nov 03 14:11:15.395811 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:15.395786 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sm2jg"] Nov 03 14:11:15.396469 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:15.396448 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61060804_c519_490a_ac90_674792d50174.slice/crio-86ece194749beba6cf9ea7331d53fffe33b4099e20a11f112133c7852f48f92c WatchSource:0}: Error finding container 86ece194749beba6cf9ea7331d53fffe33b4099e20a11f112133c7852f48f92c: Status 404 returned error can't find the container with id 86ece194749beba6cf9ea7331d53fffe33b4099e20a11f112133c7852f48f92c Nov 03 14:11:15.408161 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:15.408139 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e011fda_08e0_4cd6_9ec9_2ee7e86b8eee.slice/crio-ea539e19fea21413d7060f6481948e4f8fe520d5cbd5570921ebd7e385485eab WatchSource:0}: Error finding container ea539e19fea21413d7060f6481948e4f8fe520d5cbd5570921ebd7e385485eab: Status 404 returned error can't find the container with id ea539e19fea21413d7060f6481948e4f8fe520d5cbd5570921ebd7e385485eab Nov 03 14:11:16.132018 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:16.131993 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xkhr9" Nov 03 14:11:16.164884 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:16.164859 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:16.165085 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:16.165072 5060 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 14:11:16.375275 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:16.375193 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sm2jg" event={"ID":"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee","Type":"ContainerStarted","Data":"b0db2ca81ba77ef993203b8c409967f2371ec7ca641b17833b064d1f1560f254"} Nov 03 14:11:16.375275 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:16.375256 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sm2jg" event={"ID":"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee","Type":"ContainerStarted","Data":"0d14ba2f8bc5a1f43c46c7407b93bf7906b993196c4c4abc7bb91cc197280b05"} Nov 03 14:11:16.375524 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:16.375289 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sm2jg" event={"ID":"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee","Type":"ContainerStarted","Data":"ea539e19fea21413d7060f6481948e4f8fe520d5cbd5570921ebd7e385485eab"} Nov 03 14:11:16.376712 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:16.376682 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2bk97" event={"ID":"61060804-c519-490a-ac90-674792d50174","Type":"ContainerStarted","Data":"86ece194749beba6cf9ea7331d53fffe33b4099e20a11f112133c7852f48f92c"} Nov 03 14:11:17.389934 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:17.389902 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2bk97" event={"ID":"61060804-c519-490a-ac90-674792d50174","Type":"ContainerStarted","Data":"d59b12f3ac0e8125ff345a914c4a793e3b4aaf9a36a4713d45460846a1669425"} Nov 03 14:11:17.692377 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:17.692353 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:11:18.396567 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:18.396528 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sm2jg" event={"ID":"3e011fda-08e0-4cd6-9ec9-2ee7e86b8eee","Type":"ContainerStarted","Data":"6a3f3df5fa3e6af46e2137e27d9b9b367fa6fe246fad8f5dea4c90998fce3549"} Nov 03 14:11:18.402352 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:18.402319 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2bk97" event={"ID":"61060804-c519-490a-ac90-674792d50174","Type":"ContainerStarted","Data":"c82ad0fddb83ef154e797a957230e515af0eab0647b424cb23aa038adc2314f6"} Nov 03 14:11:18.402490 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:18.402401 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2bk97" Nov 03 14:11:20.242099 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:20.242074 5060 scope.go:122] "RemoveContainer" containerID="ecd927bb274859bd54bc1db7522266968c4926ad29d8376341481d3ad15ddaea" Nov 03 14:11:20.411590 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:20.411568 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/3.log" Nov 03 14:11:20.412036 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:20.412017 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/2.log" Nov 03 14:11:20.412150 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:20.412052 5060 generic.go:356] "Generic (PLEG): container finished" podID="307e48f6-4a66-4c9c-a4ff-848cce9d3e0e" containerID="630eac4da9d9b93c96a2686b81dd385ecade73dbd3eafdde54c06bfe820161e7" exitCode=255 Nov 03 14:11:20.412150 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:20.412108 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" event={"ID":"307e48f6-4a66-4c9c-a4ff-848cce9d3e0e","Type":"ContainerDied","Data":"630eac4da9d9b93c96a2686b81dd385ecade73dbd3eafdde54c06bfe820161e7"} Nov 03 14:11:20.412237 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:20.412194 5060 scope.go:122] "RemoveContainer" containerID="ecd927bb274859bd54bc1db7522266968c4926ad29d8376341481d3ad15ddaea" Nov 03 14:11:20.412598 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:20.412580 5060 scope.go:122] "RemoveContainer" containerID="630eac4da9d9b93c96a2686b81dd385ecade73dbd3eafdde54c06bfe820161e7" Nov 03 14:11:20.412806 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:20.412784 5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5xzwg_openshift-console-operator(307e48f6-4a66-4c9c-a4ff-848cce9d3e0e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" podUID="307e48f6-4a66-4c9c-a4ff-848cce9d3e0e" Nov 03 14:11:20.734914 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:20.727954 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" Nov 03 14:11:21.418394 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:21.418366 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/3.log" Nov 03 14:11:21.418915 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:21.418790 5060 scope.go:122] "RemoveContainer" containerID="630eac4da9d9b93c96a2686b81dd385ecade73dbd3eafdde54c06bfe820161e7" Nov 03 14:11:21.419092 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:21.419061 5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5xzwg_openshift-console-operator(307e48f6-4a66-4c9c-a4ff-848cce9d3e0e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" podUID="307e48f6-4a66-4c9c-a4ff-848cce9d3e0e" Nov 03 14:11:23.270647 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.270608 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-mq6lc-lk775"] Nov 03 14:11:23.275716 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.275689 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-mq6lc-lk775" Nov 03 14:11:23.282417 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.282383 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9bb7m"] Nov 03 14:11:23.289269 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.288370 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9bb7m" Nov 03 14:11:23.305628 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.305599 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57647df665-9q52v"] Nov 03 14:11:23.309406 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.309372 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-mq6lc-lk775"] Nov 03 14:11:23.313201 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.313176 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 03 14:11:23.314415 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.314388 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-zxg7n\"" Nov 03 14:11:23.316450 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.316399 5060 patch_prober.go:29] interesting pod/image-registry-57647df665-9q52v container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 14:11:23.316575 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.316505 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-57647df665-9q52v" podUID="fbfeab8a-b74e-4188-b565-b56442a01b79" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 14:11:23.328273 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.328242 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9bb7m"] Nov 03 14:11:23.386872 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.386817 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7c486d9667-j4szh"] Nov 03 14:11:23.390190 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.390171 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c486d9667-j4szh" Nov 03 14:11:23.407534 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.407495 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f6eebaa-adb5-4ba8-b95e-face3189c579-trusted-ca-bundle\") pod \"periodic-gathering-mq6lc-lk775\" (UID: \"8f6eebaa-adb5-4ba8-b95e-face3189c579\") " pod="openshift-insights/periodic-gathering-mq6lc-lk775" Nov 03 14:11:23.407694 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.407560 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8f6eebaa-adb5-4ba8-b95e-face3189c579-archives-path\") pod \"periodic-gathering-mq6lc-lk775\" (UID: \"8f6eebaa-adb5-4ba8-b95e-face3189c579\") " pod="openshift-insights/periodic-gathering-mq6lc-lk775" Nov 03 14:11:23.407694 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.407628 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb9d5\" (UniqueName: \"kubernetes.io/projected/8f6eebaa-adb5-4ba8-b95e-face3189c579-kube-api-access-nb9d5\") pod \"periodic-gathering-mq6lc-lk775\" (UID: \"8f6eebaa-adb5-4ba8-b95e-face3189c579\") " pod="openshift-insights/periodic-gathering-mq6lc-lk775" Nov 03 14:11:23.407694 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.407676 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f5699d04-232e-4267-9c1b-8b206a22fa11-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-9bb7m\" (UID: \"f5699d04-232e-4267-9c1b-8b206a22fa11\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9bb7m" Nov 03 14:11:23.407792 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.407710 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f6eebaa-adb5-4ba8-b95e-face3189c579-service-ca-bundle\") pod \"periodic-gathering-mq6lc-lk775\" (UID: \"8f6eebaa-adb5-4ba8-b95e-face3189c579\") " pod="openshift-insights/periodic-gathering-mq6lc-lk775" Nov 03 14:11:23.418263 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.418213 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c486d9667-j4szh"] Nov 03 14:11:23.508997 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.508964 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f6eebaa-adb5-4ba8-b95e-face3189c579-service-ca-bundle\") pod \"periodic-gathering-mq6lc-lk775\" (UID: \"8f6eebaa-adb5-4ba8-b95e-face3189c579\") " pod="openshift-insights/periodic-gathering-mq6lc-lk775" Nov 03 14:11:23.509167 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.509004 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkplg\" (UniqueName: \"kubernetes.io/projected/cd3973a6-477c-4cbf-88eb-1cf438e2bee6-kube-api-access-wkplg\") pod \"image-registry-7c486d9667-j4szh\" (UID: \"cd3973a6-477c-4cbf-88eb-1cf438e2bee6\") " pod="openshift-image-registry/image-registry-7c486d9667-j4szh" Nov 03 14:11:23.509167 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.509035 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd3973a6-477c-4cbf-88eb-1cf438e2bee6-installation-pull-secrets\") pod \"image-registry-7c486d9667-j4szh\" (UID: \"cd3973a6-477c-4cbf-88eb-1cf438e2bee6\") " pod="openshift-image-registry/image-registry-7c486d9667-j4szh" Nov 03 14:11:23.509167 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.509143 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f6eebaa-adb5-4ba8-b95e-face3189c579-trusted-ca-bundle\") pod \"periodic-gathering-mq6lc-lk775\" (UID: \"8f6eebaa-adb5-4ba8-b95e-face3189c579\") " pod="openshift-insights/periodic-gathering-mq6lc-lk775" Nov 03 14:11:23.509282 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.509182 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd3973a6-477c-4cbf-88eb-1cf438e2bee6-trusted-ca\") pod \"image-registry-7c486d9667-j4szh\" (UID: \"cd3973a6-477c-4cbf-88eb-1cf438e2bee6\") " pod="openshift-image-registry/image-registry-7c486d9667-j4szh" Nov 03 14:11:23.509282 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.509207 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd3973a6-477c-4cbf-88eb-1cf438e2bee6-image-registry-private-configuration\") pod \"image-registry-7c486d9667-j4szh\" (UID: \"cd3973a6-477c-4cbf-88eb-1cf438e2bee6\") " pod="openshift-image-registry/image-registry-7c486d9667-j4szh" Nov 03 14:11:23.509356 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.509300 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd3973a6-477c-4cbf-88eb-1cf438e2bee6-registry-tls\") pod \"image-registry-7c486d9667-j4szh\" (UID: \"cd3973a6-477c-4cbf-88eb-1cf438e2bee6\") " pod="openshift-image-registry/image-registry-7c486d9667-j4szh" Nov 03 14:11:23.509356 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.509340 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8f6eebaa-adb5-4ba8-b95e-face3189c579-archives-path\") pod \"periodic-gathering-mq6lc-lk775\" (UID: \"8f6eebaa-adb5-4ba8-b95e-face3189c579\") " pod="openshift-insights/periodic-gathering-mq6lc-lk775" Nov 03 14:11:23.509447 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.509369 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb9d5\" (UniqueName: \"kubernetes.io/projected/8f6eebaa-adb5-4ba8-b95e-face3189c579-kube-api-access-nb9d5\") pod \"periodic-gathering-mq6lc-lk775\" (UID: \"8f6eebaa-adb5-4ba8-b95e-face3189c579\") " pod="openshift-insights/periodic-gathering-mq6lc-lk775" Nov 03 14:11:23.509447 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.509402 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd3973a6-477c-4cbf-88eb-1cf438e2bee6-ca-trust-extracted\") pod \"image-registry-7c486d9667-j4szh\" (UID: \"cd3973a6-477c-4cbf-88eb-1cf438e2bee6\") " pod="openshift-image-registry/image-registry-7c486d9667-j4szh" Nov 03 14:11:23.509447 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.509427 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd3973a6-477c-4cbf-88eb-1cf438e2bee6-registry-certificates\") pod \"image-registry-7c486d9667-j4szh\" (UID: \"cd3973a6-477c-4cbf-88eb-1cf438e2bee6\") " pod="openshift-image-registry/image-registry-7c486d9667-j4szh" Nov 03 14:11:23.509570 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.509542 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd3973a6-477c-4cbf-88eb-1cf438e2bee6-bound-sa-token\") pod \"image-registry-7c486d9667-j4szh\" (UID: \"cd3973a6-477c-4cbf-88eb-1cf438e2bee6\") " pod="openshift-image-registry/image-registry-7c486d9667-j4szh" Nov 03 14:11:23.509629 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.509614 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f5699d04-232e-4267-9c1b-8b206a22fa11-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-9bb7m\" (UID: \"f5699d04-232e-4267-9c1b-8b206a22fa11\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9bb7m" Nov 03 14:11:23.509773 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.509757 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8f6eebaa-adb5-4ba8-b95e-face3189c579-archives-path\") pod \"periodic-gathering-mq6lc-lk775\" (UID: \"8f6eebaa-adb5-4ba8-b95e-face3189c579\") " pod="openshift-insights/periodic-gathering-mq6lc-lk775" Nov 03 14:11:23.509828 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.509774 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f6eebaa-adb5-4ba8-b95e-face3189c579-service-ca-bundle\") pod \"periodic-gathering-mq6lc-lk775\" (UID: \"8f6eebaa-adb5-4ba8-b95e-face3189c579\") " pod="openshift-insights/periodic-gathering-mq6lc-lk775" Nov 03 14:11:23.510440 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.510420 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f6eebaa-adb5-4ba8-b95e-face3189c579-trusted-ca-bundle\") pod \"periodic-gathering-mq6lc-lk775\" (UID: \"8f6eebaa-adb5-4ba8-b95e-face3189c579\") " pod="openshift-insights/periodic-gathering-mq6lc-lk775" Nov 03 14:11:23.512022 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.512001 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f5699d04-232e-4267-9c1b-8b206a22fa11-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-9bb7m\" (UID: \"f5699d04-232e-4267-9c1b-8b206a22fa11\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9bb7m" Nov 03 14:11:23.531529 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.531475 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb9d5\" (UniqueName: \"kubernetes.io/projected/8f6eebaa-adb5-4ba8-b95e-face3189c579-kube-api-access-nb9d5\") pod \"periodic-gathering-mq6lc-lk775\" (UID: \"8f6eebaa-adb5-4ba8-b95e-face3189c579\") " pod="openshift-insights/periodic-gathering-mq6lc-lk775" Nov 03 14:11:23.598881 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.598828 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-mq6lc-lk775" Nov 03 14:11:23.610030 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.610002 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd3973a6-477c-4cbf-88eb-1cf438e2bee6-ca-trust-extracted\") pod \"image-registry-7c486d9667-j4szh\" (UID: \"cd3973a6-477c-4cbf-88eb-1cf438e2bee6\") " pod="openshift-image-registry/image-registry-7c486d9667-j4szh" Nov 03 14:11:23.610162 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.610036 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd3973a6-477c-4cbf-88eb-1cf438e2bee6-registry-certificates\") pod \"image-registry-7c486d9667-j4szh\" (UID: \"cd3973a6-477c-4cbf-88eb-1cf438e2bee6\") " pod="openshift-image-registry/image-registry-7c486d9667-j4szh" Nov 03 14:11:23.610162 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.610060 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd3973a6-477c-4cbf-88eb-1cf438e2bee6-bound-sa-token\") pod \"image-registry-7c486d9667-j4szh\" (UID: \"cd3973a6-477c-4cbf-88eb-1cf438e2bee6\") " pod="openshift-image-registry/image-registry-7c486d9667-j4szh" Nov 03 14:11:23.610162 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.610100 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkplg\" (UniqueName: \"kubernetes.io/projected/cd3973a6-477c-4cbf-88eb-1cf438e2bee6-kube-api-access-wkplg\") pod \"image-registry-7c486d9667-j4szh\" (UID: \"cd3973a6-477c-4cbf-88eb-1cf438e2bee6\") " pod="openshift-image-registry/image-registry-7c486d9667-j4szh" Nov 03 14:11:23.610162 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.610128 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd3973a6-477c-4cbf-88eb-1cf438e2bee6-installation-pull-secrets\") pod \"image-registry-7c486d9667-j4szh\" (UID: \"cd3973a6-477c-4cbf-88eb-1cf438e2bee6\") " pod="openshift-image-registry/image-registry-7c486d9667-j4szh" Nov 03 14:11:23.610361 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.610179 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd3973a6-477c-4cbf-88eb-1cf438e2bee6-trusted-ca\") pod \"image-registry-7c486d9667-j4szh\" (UID: \"cd3973a6-477c-4cbf-88eb-1cf438e2bee6\") " pod="openshift-image-registry/image-registry-7c486d9667-j4szh" Nov 03 14:11:23.610361 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.610206 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd3973a6-477c-4cbf-88eb-1cf438e2bee6-image-registry-private-configuration\") pod \"image-registry-7c486d9667-j4szh\" (UID: \"cd3973a6-477c-4cbf-88eb-1cf438e2bee6\") " pod="openshift-image-registry/image-registry-7c486d9667-j4szh" Nov 03 14:11:23.610361 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.610253 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd3973a6-477c-4cbf-88eb-1cf438e2bee6-registry-tls\") pod \"image-registry-7c486d9667-j4szh\" (UID: \"cd3973a6-477c-4cbf-88eb-1cf438e2bee6\") " pod="openshift-image-registry/image-registry-7c486d9667-j4szh" Nov 03 14:11:23.611005 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.610973 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd3973a6-477c-4cbf-88eb-1cf438e2bee6-registry-certificates\") pod \"image-registry-7c486d9667-j4szh\" (UID: \"cd3973a6-477c-4cbf-88eb-1cf438e2bee6\") " pod="openshift-image-registry/image-registry-7c486d9667-j4szh" Nov 03 14:11:23.611317 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.611284 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd3973a6-477c-4cbf-88eb-1cf438e2bee6-trusted-ca\") pod \"image-registry-7c486d9667-j4szh\" (UID: \"cd3973a6-477c-4cbf-88eb-1cf438e2bee6\") " pod="openshift-image-registry/image-registry-7c486d9667-j4szh" Nov 03 14:11:23.611896 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.611855 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd3973a6-477c-4cbf-88eb-1cf438e2bee6-ca-trust-extracted\") pod \"image-registry-7c486d9667-j4szh\" (UID: \"cd3973a6-477c-4cbf-88eb-1cf438e2bee6\") " pod="openshift-image-registry/image-registry-7c486d9667-j4szh" Nov 03 14:11:23.612947 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.612923 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd3973a6-477c-4cbf-88eb-1cf438e2bee6-installation-pull-secrets\") pod \"image-registry-7c486d9667-j4szh\" (UID: \"cd3973a6-477c-4cbf-88eb-1cf438e2bee6\") " pod="openshift-image-registry/image-registry-7c486d9667-j4szh" Nov 03 14:11:23.613069 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.613038 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd3973a6-477c-4cbf-88eb-1cf438e2bee6-image-registry-private-configuration\") pod \"image-registry-7c486d9667-j4szh\" (UID: \"cd3973a6-477c-4cbf-88eb-1cf438e2bee6\") " pod="openshift-image-registry/image-registry-7c486d9667-j4szh" Nov 03 14:11:23.613540 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.613524 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd3973a6-477c-4cbf-88eb-1cf438e2bee6-registry-tls\") pod \"image-registry-7c486d9667-j4szh\" (UID: \"cd3973a6-477c-4cbf-88eb-1cf438e2bee6\") " pod="openshift-image-registry/image-registry-7c486d9667-j4szh" Nov 03 14:11:23.619424 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.619403 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9bb7m" Nov 03 14:11:23.632450 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.632427 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd3973a6-477c-4cbf-88eb-1cf438e2bee6-bound-sa-token\") pod \"image-registry-7c486d9667-j4szh\" (UID: \"cd3973a6-477c-4cbf-88eb-1cf438e2bee6\") " pod="openshift-image-registry/image-registry-7c486d9667-j4szh" Nov 03 14:11:23.632572 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.632455 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkplg\" (UniqueName: \"kubernetes.io/projected/cd3973a6-477c-4cbf-88eb-1cf438e2bee6-kube-api-access-wkplg\") pod \"image-registry-7c486d9667-j4szh\" (UID: \"cd3973a6-477c-4cbf-88eb-1cf438e2bee6\") " pod="openshift-image-registry/image-registry-7c486d9667-j4szh" Nov 03 14:11:23.702307 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.702265 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c486d9667-j4szh" Nov 03 14:11:23.900278 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.900207 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-mq6lc-lk775"] Nov 03 14:11:23.903553 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:23.903524 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f6eebaa_adb5_4ba8_b95e_face3189c579.slice/crio-c628c9690ce6dd066f0d8feacff748142ecd0088225a7bdbc912352e5c89f7d7 WatchSource:0}: Error finding container c628c9690ce6dd066f0d8feacff748142ecd0088225a7bdbc912352e5c89f7d7: Status 404 returned error can't find the container with id c628c9690ce6dd066f0d8feacff748142ecd0088225a7bdbc912352e5c89f7d7 Nov 03 14:11:23.926061 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:23.926040 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9bb7m"] Nov 03 14:11:23.936892 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:23.936868 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5699d04_232e_4267_9c1b_8b206a22fa11.slice/crio-23423107f62eb12087406e1c4eeb2577cfdefab51c9fc4751b219f45071ee622 WatchSource:0}: Error finding container 23423107f62eb12087406e1c4eeb2577cfdefab51c9fc4751b219f45071ee622: Status 404 returned error can't find the container with id 23423107f62eb12087406e1c4eeb2577cfdefab51c9fc4751b219f45071ee622 Nov 03 14:11:24.004523 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:24.004491 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c486d9667-j4szh"] Nov 03 14:11:24.008966 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:24.008933 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd3973a6_477c_4cbf_88eb_1cf438e2bee6.slice/crio-cbe71a8b9bcfa19cd29b431fbe211121526e7a4924d3ee1122b9c30be8814e96 WatchSource:0}: Error finding container cbe71a8b9bcfa19cd29b431fbe211121526e7a4924d3ee1122b9c30be8814e96: Status 404 returned error can't find the container with id cbe71a8b9bcfa19cd29b431fbe211121526e7a4924d3ee1122b9c30be8814e96 Nov 03 14:11:24.433415 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:24.433376 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c486d9667-j4szh" event={"ID":"cd3973a6-477c-4cbf-88eb-1cf438e2bee6","Type":"ContainerStarted","Data":"0289e2b1c217aed21c66ae3cb827b9435d5d4d31fbcb04b7bb18c326c43fbe87"} Nov 03 14:11:24.433953 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:24.433433 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c486d9667-j4szh" event={"ID":"cd3973a6-477c-4cbf-88eb-1cf438e2bee6","Type":"ContainerStarted","Data":"cbe71a8b9bcfa19cd29b431fbe211121526e7a4924d3ee1122b9c30be8814e96"} Nov 03 14:11:24.433953 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:24.433490 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7c486d9667-j4szh" Nov 03 14:11:24.435014 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:24.434977 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9bb7m" event={"ID":"f5699d04-232e-4267-9c1b-8b206a22fa11","Type":"ContainerStarted","Data":"23423107f62eb12087406e1c4eeb2577cfdefab51c9fc4751b219f45071ee622"} Nov 03 14:11:24.436714 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:24.436692 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-mq6lc-lk775" event={"ID":"8f6eebaa-adb5-4ba8-b95e-face3189c579","Type":"ContainerStarted","Data":"1f632f132a553777eefd5caa2880f6dfbdd59aa7d4f3ebf7365968779ddb5844"} Nov 03 14:11:24.436823 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:24.436728 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-mq6lc-lk775" event={"ID":"8f6eebaa-adb5-4ba8-b95e-face3189c579","Type":"ContainerStarted","Data":"c628c9690ce6dd066f0d8feacff748142ecd0088225a7bdbc912352e5c89f7d7"} Nov 03 14:11:24.480157 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:24.480093 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7c486d9667-j4szh" podStartSLOduration=1.480072137 podStartE2EDuration="1.480072137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:11:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:24.477795535 +0000 UTC m=+18.366585116" watchObservedRunningTime="2026-11-03 14:11:24.480072137 +0000 UTC m=+18.368861718" Nov 03 14:11:24.529387 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:24.529329 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-mq6lc-lk775" podStartSLOduration=1.5293142400000002 podStartE2EDuration="1.52931424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:11:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:24.528254098 +0000 UTC m=+18.417043681" watchObservedRunningTime="2026-11-03 14:11:24.52931424 +0000 UTC m=+18.418103820" Nov 03 14:11:24.626117 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:24.626084 5060 patch_prober.go:29] interesting pod/image-registry-6d7c596f48-qrcsg container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 14:11:24.626285 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:24.626141 5060 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" podUID="b3260ef2-8ff4-4b70-a3a0-e5386a27d210" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 14:11:24.912859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:24.912814 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2bk97_61060804-c519-490a-ac90-674792d50174/dns/0.log" Nov 03 14:11:25.112312 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:25.112287 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2bk97_61060804-c519-490a-ac90-674792d50174/kube-rbac-proxy/0.log" Nov 03 14:11:25.117750 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:25.117723 5060 patch_prober.go:29] interesting pod/image-registry-6d7c596f48-qrcsg container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 14:11:25.117898 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:25.117769 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" podUID="b3260ef2-8ff4-4b70-a3a0-e5386a27d210" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 14:11:25.442078 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:25.441983 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9bb7m" event={"ID":"f5699d04-232e-4267-9c1b-8b206a22fa11","Type":"ContainerStarted","Data":"8312a59e49b4ea02c8ebd6c822f72204caecb685946e428c258221e3cf9bd6ae"} Nov 03 14:11:25.442078 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:25.442050 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9bb7m" Nov 03 14:11:25.451088 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:25.451059 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9bb7m" Nov 03 14:11:25.485245 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:25.485189 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9bb7m" podStartSLOduration=1.510526523 podStartE2EDuration="2.485177591s" totalImagesPullingTime="974.651068ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:11:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:25.484809877 +0000 UTC m=+19.373599457" watchObservedRunningTime="2026-11-03 14:11:25.485177591 +0000 UTC m=+19.373967165" Nov 03 14:11:25.512460 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:25.512437 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q4gsb_e637e650-5fc6-4adb-a97a-47d924c30f97/dns-node-resolver/0.log" Nov 03 14:11:25.912978 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:25.912950 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-57647df665-9q52v_fbfeab8a-b74e-4188-b565-b56442a01b79/registry/0.log" Nov 03 14:11:26.113239 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:26.113192 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6d7c596f48-qrcsg_b3260ef2-8ff4-4b70-a3a0-e5386a27d210/registry/0.log" Nov 03 14:11:26.316862 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:26.316810 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7c486d9667-j4szh_cd3973a6-477c-4cbf-88eb-1cf438e2bee6/registry/0.log" Nov 03 14:11:26.349155 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:26.349123 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-dh22t"] Nov 03 14:11:26.352625 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:26.352609 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-dh22t" Nov 03 14:11:26.381933 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:26.381901 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 03 14:11:26.382201 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:26.382177 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 03 14:11:26.382351 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:26.382213 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-hx7hd\"" Nov 03 14:11:26.383089 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:26.383071 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 14:11:26.384472 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:26.384446 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-dh22t"] Nov 03 14:11:26.451390 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:26.451353 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptqtf\" (UniqueName: \"kubernetes.io/projected/aea82995-1556-4f76-a701-53146ad84381-kube-api-access-ptqtf\") pod \"prometheus-operator-9cd89b74c-dh22t\" (UID: \"aea82995-1556-4f76-a701-53146ad84381\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dh22t" Nov 03 14:11:26.451905 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:26.451442 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aea82995-1556-4f76-a701-53146ad84381-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-dh22t\" (UID: \"aea82995-1556-4f76-a701-53146ad84381\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dh22t" Nov 03 14:11:26.451905 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:26.451496 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aea82995-1556-4f76-a701-53146ad84381-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-dh22t\" (UID: \"aea82995-1556-4f76-a701-53146ad84381\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dh22t" Nov 03 14:11:26.451905 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:26.451567 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/aea82995-1556-4f76-a701-53146ad84381-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-dh22t\" (UID: \"aea82995-1556-4f76-a701-53146ad84381\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dh22t" Nov 03 14:11:26.552467 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:26.552434 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aea82995-1556-4f76-a701-53146ad84381-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-dh22t\" (UID: \"aea82995-1556-4f76-a701-53146ad84381\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dh22t" Nov 03 14:11:26.552639 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:26.552473 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/aea82995-1556-4f76-a701-53146ad84381-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-dh22t\" (UID: \"aea82995-1556-4f76-a701-53146ad84381\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dh22t" Nov 03 14:11:26.552639 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:26.552504 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptqtf\" (UniqueName: \"kubernetes.io/projected/aea82995-1556-4f76-a701-53146ad84381-kube-api-access-ptqtf\") pod \"prometheus-operator-9cd89b74c-dh22t\" (UID: \"aea82995-1556-4f76-a701-53146ad84381\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dh22t" Nov 03 14:11:26.552639 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:26.552585 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aea82995-1556-4f76-a701-53146ad84381-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-dh22t\" (UID: \"aea82995-1556-4f76-a701-53146ad84381\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dh22t" Nov 03 14:11:26.553202 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:26.553180 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aea82995-1556-4f76-a701-53146ad84381-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-dh22t\" (UID: \"aea82995-1556-4f76-a701-53146ad84381\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dh22t" Nov 03 14:11:26.554902 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:26.554881 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aea82995-1556-4f76-a701-53146ad84381-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-dh22t\" (UID: \"aea82995-1556-4f76-a701-53146ad84381\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dh22t" Nov 03 14:11:26.554986 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:26.554888 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/aea82995-1556-4f76-a701-53146ad84381-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-dh22t\" (UID: \"aea82995-1556-4f76-a701-53146ad84381\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dh22t" Nov 03 14:11:26.579588 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:26.579531 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptqtf\" (UniqueName: \"kubernetes.io/projected/aea82995-1556-4f76-a701-53146ad84381-kube-api-access-ptqtf\") pod \"prometheus-operator-9cd89b74c-dh22t\" (UID: \"aea82995-1556-4f76-a701-53146ad84381\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dh22t" Nov 03 14:11:26.665588 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:26.665553 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-dh22t" Nov 03 14:11:26.912075 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:26.911996 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t4b94_e68be0c4-d4ee-4da5-82e3-82372f811841/node-ca/0.log" Nov 03 14:11:26.961994 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:26.961963 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-dh22t"] Nov 03 14:11:26.964916 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:26.964885 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaea82995_1556_4f76_a701_53146ad84381.slice/crio-db47cc7bb5b3b0c89cccde68f19c0d05ef0e1441bdfb5a51c9e457c62d617f77 WatchSource:0}: Error finding container db47cc7bb5b3b0c89cccde68f19c0d05ef0e1441bdfb5a51c9e457c62d617f77: Status 404 returned error can't find the container with id db47cc7bb5b3b0c89cccde68f19c0d05ef0e1441bdfb5a51c9e457c62d617f77 Nov 03 14:11:26.972210 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:26.972183 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6d6z" Nov 03 14:11:27.124268 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:27.124239 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6c7f54c9b7-t6wbf_36065516-87ab-4057-b198-439305606d12/router/0.log" Nov 03 14:11:27.312112 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:27.312083 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qnjdm_63bfb9d0-9e87-44c6-a6f7-2f7b992e20f6/serve-healthcheck-canary/0.log" Nov 03 14:11:27.454595 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:27.454559 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-dh22t" event={"ID":"aea82995-1556-4f76-a701-53146ad84381","Type":"ContainerStarted","Data":"db47cc7bb5b3b0c89cccde68f19c0d05ef0e1441bdfb5a51c9e457c62d617f77"} Nov 03 14:11:28.407828 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:28.407757 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2bk97" Nov 03 14:11:28.460942 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:28.460905 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-dh22t" event={"ID":"aea82995-1556-4f76-a701-53146ad84381","Type":"ContainerStarted","Data":"cbbace1cdac6c8b9427e269001506c0c03c9cbfad0bc44304c00e48a516cbd0c"} Nov 03 14:11:28.461414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:28.460966 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-dh22t" event={"ID":"aea82995-1556-4f76-a701-53146ad84381","Type":"ContainerStarted","Data":"27d1fbf5559bb617eef20e7077c08a718e677445083b964680942107a2630c95"} Nov 03 14:11:28.508868 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:28.508804 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-dh22t" podStartSLOduration=1.380550162 podStartE2EDuration="2.508790917s" totalImagesPullingTime="1.128240755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:11:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:28.507414586 +0000 UTC m=+22.396204169" watchObservedRunningTime="2026-11-03 14:11:28.508790917 +0000 UTC m=+22.397580497" Nov 03 14:11:28.861772 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:28.861735 5060 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" Nov 03 14:11:28.862208 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:28.862192 5060 scope.go:122] "RemoveContainer" containerID="630eac4da9d9b93c96a2686b81dd385ecade73dbd3eafdde54c06bfe820161e7" Nov 03 14:11:28.862408 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:28.862389 5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5xzwg_openshift-console-operator(307e48f6-4a66-4c9c-a4ff-848cce9d3e0e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" podUID="307e48f6-4a66-4c9c-a4ff-848cce9d3e0e" Nov 03 14:11:31.125540 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:31.125501 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-znvl4"] Nov 03 14:11:31.165469 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:31.165438 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-znvl4" Nov 03 14:11:31.184990 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:31.184962 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/95d09114-670c-4ca3-bbcc-f41b5f8464e1-root\") pod \"node-exporter-znvl4\" (UID: \"95d09114-670c-4ca3-bbcc-f41b5f8464e1\") " pod="openshift-monitoring/node-exporter-znvl4" Nov 03 14:11:31.185132 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:31.185077 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/95d09114-670c-4ca3-bbcc-f41b5f8464e1-node-exporter-wtmp\") pod \"node-exporter-znvl4\" (UID: \"95d09114-670c-4ca3-bbcc-f41b5f8464e1\") " pod="openshift-monitoring/node-exporter-znvl4" Nov 03 14:11:31.185132 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:31.185113 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/95d09114-670c-4ca3-bbcc-f41b5f8464e1-node-exporter-textfile\") pod \"node-exporter-znvl4\" (UID: \"95d09114-670c-4ca3-bbcc-f41b5f8464e1\") " pod="openshift-monitoring/node-exporter-znvl4" Nov 03 14:11:31.185222 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:31.185141 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/95d09114-670c-4ca3-bbcc-f41b5f8464e1-node-exporter-tls\") pod \"node-exporter-znvl4\" (UID: \"95d09114-670c-4ca3-bbcc-f41b5f8464e1\") " pod="openshift-monitoring/node-exporter-znvl4" Nov 03 14:11:31.185222 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:31.185204 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95d09114-670c-4ca3-bbcc-f41b5f8464e1-metrics-client-ca\") pod \"node-exporter-znvl4\" (UID: \"95d09114-670c-4ca3-bbcc-f41b5f8464e1\") " pod="openshift-monitoring/node-exporter-znvl4" Nov 03 14:11:31.185305 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:31.185234 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42jjv\" (UniqueName: \"kubernetes.io/projected/95d09114-670c-4ca3-bbcc-f41b5f8464e1-kube-api-access-42jjv\") pod \"node-exporter-znvl4\" (UID: \"95d09114-670c-4ca3-bbcc-f41b5f8464e1\") " pod="openshift-monitoring/node-exporter-znvl4" Nov 03 14:11:31.185305 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:31.185255 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/95d09114-670c-4ca3-bbcc-f41b5f8464e1-sys\") pod \"node-exporter-znvl4\" (UID: \"95d09114-670c-4ca3-bbcc-f41b5f8464e1\") " pod="openshift-monitoring/node-exporter-znvl4" Nov 03 14:11:31.185305 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:31.185274 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/95d09114-670c-4ca3-bbcc-f41b5f8464e1-node-exporter-accelerators-collector-config\") pod \"node-exporter-znvl4\" (UID: \"95d09114-670c-4ca3-bbcc-f41b5f8464e1\") " pod="openshift-monitoring/node-exporter-znvl4" Nov 03 14:11:31.185452 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:31.185402 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/95d09114-670c-4ca3-bbcc-f41b5f8464e1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-znvl4\" (UID: \"95d09114-670c-4ca3-bbcc-f41b5f8464e1\") " pod="openshift-monitoring/node-exporter-znvl4" Nov 03 14:11:31.251570 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:31.251542 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 14:11:31.256949 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:31.256918 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 14:11:31.257111 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:31.257078 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zjlcb\"" Nov 03 14:11:31.257174 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:31.257163 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 14:11:31.286539 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:31.286505 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/95d09114-670c-4ca3-bbcc-f41b5f8464e1-sys\") pod \"node-exporter-znvl4\" (UID: \"95d09114-670c-4ca3-bbcc-f41b5f8464e1\") " pod="openshift-monitoring/node-exporter-znvl4" Nov 03 14:11:31.286539 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:31.286542 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/95d09114-670c-4ca3-bbcc-f41b5f8464e1-node-exporter-accelerators-collector-config\") pod \"node-exporter-znvl4\" (UID: \"95d09114-670c-4ca3-bbcc-f41b5f8464e1\") " pod="openshift-monitoring/node-exporter-znvl4" Nov 03 14:11:31.286739 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:31.286565 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/95d09114-670c-4ca3-bbcc-f41b5f8464e1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-znvl4\" (UID: \"95d09114-670c-4ca3-bbcc-f41b5f8464e1\") " pod="openshift-monitoring/node-exporter-znvl4" Nov 03 14:11:31.286739 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:31.286582 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/95d09114-670c-4ca3-bbcc-f41b5f8464e1-root\") pod \"node-exporter-znvl4\" (UID: \"95d09114-670c-4ca3-bbcc-f41b5f8464e1\") " pod="openshift-monitoring/node-exporter-znvl4" Nov 03 14:11:31.286739 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:31.286631 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/95d09114-670c-4ca3-bbcc-f41b5f8464e1-node-exporter-wtmp\") pod \"node-exporter-znvl4\" (UID: \"95d09114-670c-4ca3-bbcc-f41b5f8464e1\") " pod="openshift-monitoring/node-exporter-znvl4" Nov 03 14:11:31.286739 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:31.286631 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/95d09114-670c-4ca3-bbcc-f41b5f8464e1-sys\") pod \"node-exporter-znvl4\" (UID: \"95d09114-670c-4ca3-bbcc-f41b5f8464e1\") " pod="openshift-monitoring/node-exporter-znvl4" Nov 03 14:11:31.286739 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:31.286652 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/95d09114-670c-4ca3-bbcc-f41b5f8464e1-node-exporter-textfile\") pod \"node-exporter-znvl4\" (UID: \"95d09114-670c-4ca3-bbcc-f41b5f8464e1\") " pod="openshift-monitoring/node-exporter-znvl4" Nov 03 14:11:31.286739 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:31.286697 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/95d09114-670c-4ca3-bbcc-f41b5f8464e1-node-exporter-tls\") pod \"node-exporter-znvl4\" (UID: \"95d09114-670c-4ca3-bbcc-f41b5f8464e1\") " pod="openshift-monitoring/node-exporter-znvl4" Nov 03 14:11:31.286739 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:31.286709 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/95d09114-670c-4ca3-bbcc-f41b5f8464e1-root\") pod \"node-exporter-znvl4\" (UID: \"95d09114-670c-4ca3-bbcc-f41b5f8464e1\") " pod="openshift-monitoring/node-exporter-znvl4" Nov 03 14:11:31.286739 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:31.286733 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95d09114-670c-4ca3-bbcc-f41b5f8464e1-metrics-client-ca\") pod \"node-exporter-znvl4\" (UID: \"95d09114-670c-4ca3-bbcc-f41b5f8464e1\") " pod="openshift-monitoring/node-exporter-znvl4" Nov 03 14:11:31.287159 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:31.286759 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42jjv\" (UniqueName: \"kubernetes.io/projected/95d09114-670c-4ca3-bbcc-f41b5f8464e1-kube-api-access-42jjv\") pod \"node-exporter-znvl4\" (UID: \"95d09114-670c-4ca3-bbcc-f41b5f8464e1\") " pod="openshift-monitoring/node-exporter-znvl4" Nov 03 14:11:31.287159 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:31.286803 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/95d09114-670c-4ca3-bbcc-f41b5f8464e1-node-exporter-wtmp\") pod \"node-exporter-znvl4\" (UID: \"95d09114-670c-4ca3-bbcc-f41b5f8464e1\") " pod="openshift-monitoring/node-exporter-znvl4" Nov 03 14:11:31.287159 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:31.287035 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/95d09114-670c-4ca3-bbcc-f41b5f8464e1-node-exporter-textfile\") pod \"node-exporter-znvl4\" (UID: \"95d09114-670c-4ca3-bbcc-f41b5f8464e1\") " pod="openshift-monitoring/node-exporter-znvl4" Nov 03 14:11:31.287258 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:31.287245 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/95d09114-670c-4ca3-bbcc-f41b5f8464e1-node-exporter-accelerators-collector-config\") pod \"node-exporter-znvl4\" (UID: \"95d09114-670c-4ca3-bbcc-f41b5f8464e1\") " pod="openshift-monitoring/node-exporter-znvl4" Nov 03 14:11:31.287342 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:31.287320 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95d09114-670c-4ca3-bbcc-f41b5f8464e1-metrics-client-ca\") pod \"node-exporter-znvl4\" (UID: \"95d09114-670c-4ca3-bbcc-f41b5f8464e1\") " pod="openshift-monitoring/node-exporter-znvl4" Nov 03 14:11:31.289086 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:31.289058 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/95d09114-670c-4ca3-bbcc-f41b5f8464e1-node-exporter-tls\") pod \"node-exporter-znvl4\" (UID: \"95d09114-670c-4ca3-bbcc-f41b5f8464e1\") " pod="openshift-monitoring/node-exporter-znvl4" Nov 03 14:11:31.289201 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:31.289182 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/95d09114-670c-4ca3-bbcc-f41b5f8464e1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-znvl4\" (UID: \"95d09114-670c-4ca3-bbcc-f41b5f8464e1\") " pod="openshift-monitoring/node-exporter-znvl4" Nov 03 14:11:31.310963 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:31.310936 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42jjv\" (UniqueName: \"kubernetes.io/projected/95d09114-670c-4ca3-bbcc-f41b5f8464e1-kube-api-access-42jjv\") pod \"node-exporter-znvl4\" (UID: \"95d09114-670c-4ca3-bbcc-f41b5f8464e1\") " pod="openshift-monitoring/node-exporter-znvl4" Nov 03 14:11:31.537389 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:31.537359 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-znvl4" Nov 03 14:11:31.575315 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:11:31.575285 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95d09114_670c_4ca3_bbcc_f41b5f8464e1.slice/crio-7aceb50edffe5080e4d940e359da74bbf995498fa8e082fc75edffb9fe61f6f6 WatchSource:0}: Error finding container 7aceb50edffe5080e4d940e359da74bbf995498fa8e082fc75edffb9fe61f6f6: Status 404 returned error can't find the container with id 7aceb50edffe5080e4d940e359da74bbf995498fa8e082fc75edffb9fe61f6f6 Nov 03 14:11:32.479130 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:32.479091 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-znvl4" event={"ID":"95d09114-670c-4ca3-bbcc-f41b5f8464e1","Type":"ContainerStarted","Data":"7aceb50edffe5080e4d940e359da74bbf995498fa8e082fc75edffb9fe61f6f6"} Nov 03 14:11:33.312784 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:33.312712 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:11:33.484918 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:33.484886 5060 generic.go:356] "Generic (PLEG): container finished" podID="95d09114-670c-4ca3-bbcc-f41b5f8464e1" containerID="27b3eee9fea61706054f548cb08421450e0cff8911099e88e6e26988748f03c8" exitCode=0 Nov 03 14:11:33.485308 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:33.484975 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-znvl4" event={"ID":"95d09114-670c-4ca3-bbcc-f41b5f8464e1","Type":"ContainerDied","Data":"27b3eee9fea61706054f548cb08421450e0cff8911099e88e6e26988748f03c8"} Nov 03 14:11:34.494226 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:34.494183 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-znvl4" event={"ID":"95d09114-670c-4ca3-bbcc-f41b5f8464e1","Type":"ContainerStarted","Data":"631ff99d5945f1cacf8bffc2a37b168e2dec03d3565da0aeb9894653bf68cfef"} Nov 03 14:11:34.494611 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:34.494249 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-znvl4" event={"ID":"95d09114-670c-4ca3-bbcc-f41b5f8464e1","Type":"ContainerStarted","Data":"0e243981b96263492a6682036111aa24d7367e9a006ef46f88e124af535961d3"} Nov 03 14:11:34.556026 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:34.555980 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-znvl4" podStartSLOduration=2.155301315 podStartE2EDuration="3.555966149s" totalImagesPullingTime="1.320601391s" totalInitContainerRuntime="80.063443ms" isStatefulPod=true podCreationTimestamp="2026-11-03 14:11:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:34.55568325 +0000 UTC m=+28.444472830" watchObservedRunningTime="2026-11-03 14:11:34.555966149 +0000 UTC m=+28.444755729" Nov 03 14:11:35.118675 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:35.118644 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:11:36.134911 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:36.134878 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xkhr9" Nov 03 14:11:39.438693 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:39.438653 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:11:41.241960 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:41.241927 5060 scope.go:122] "RemoveContainer" containerID="630eac4da9d9b93c96a2686b81dd385ecade73dbd3eafdde54c06bfe820161e7" Nov 03 14:11:41.527858 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:41.527775 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/3.log" Nov 03 14:11:41.528019 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:41.527872 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" event={"ID":"307e48f6-4a66-4c9c-a4ff-848cce9d3e0e","Type":"ContainerStarted","Data":"fc86e845fa5974bd0bdeaf2a303ac013bbde0b0c0629740a3b05a1aeb68e12a5"} Nov 03 14:11:41.528177 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:41.528157 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" Nov 03 14:11:41.744868 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:41.744821 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-5xzwg" Nov 03 14:11:45.447395 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:45.447363 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7c486d9667-j4szh" Nov 03 14:11:45.554879 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:45.554847 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d7c596f48-qrcsg"] Nov 03 14:11:48.341268 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:48.341159 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-57647df665-9q52v" podUID="fbfeab8a-b74e-4188-b565-b56442a01b79" containerName="registry" containerID="cri-o://af30d755532f37de5e52d15dc1fb476952fd52dc3c1bcec2b2e7c2700d5a4763" gracePeriod=30 Nov 03 14:11:48.564748 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:48.564715 5060 generic.go:356] "Generic (PLEG): container finished" podID="fbfeab8a-b74e-4188-b565-b56442a01b79" containerID="af30d755532f37de5e52d15dc1fb476952fd52dc3c1bcec2b2e7c2700d5a4763" exitCode=0 Nov 03 14:11:48.564953 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:48.564783 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57647df665-9q52v" event={"ID":"fbfeab8a-b74e-4188-b565-b56442a01b79","Type":"ContainerDied","Data":"af30d755532f37de5e52d15dc1fb476952fd52dc3c1bcec2b2e7c2700d5a4763"} Nov 03 14:11:48.599763 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:48.599700 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:11:48.657752 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:48.657718 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbfeab8a-b74e-4188-b565-b56442a01b79-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fbfeab8a-b74e-4188-b565-b56442a01b79-registry-certificates\") pod \"fbfeab8a-b74e-4188-b565-b56442a01b79\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " Nov 03 14:11:48.657955 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:48.657767 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbfeab8a-b74e-4188-b565-b56442a01b79-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fbfeab8a-b74e-4188-b565-b56442a01b79-ca-trust-extracted\") pod \"fbfeab8a-b74e-4188-b565-b56442a01b79\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " Nov 03 14:11:48.657955 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:48.657797 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbfeab8a-b74e-4188-b565-b56442a01b79-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fbfeab8a-b74e-4188-b565-b56442a01b79-installation-pull-secrets\") pod \"fbfeab8a-b74e-4188-b565-b56442a01b79\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " Nov 03 14:11:48.657955 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:48.657887 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-bound-sa-token\") pod \"fbfeab8a-b74e-4188-b565-b56442a01b79\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " Nov 03 14:11:48.657955 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:48.657912 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbfeab8a-b74e-4188-b565-b56442a01b79-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fbfeab8a-b74e-4188-b565-b56442a01b79-trusted-ca\") pod \"fbfeab8a-b74e-4188-b565-b56442a01b79\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " Nov 03 14:11:48.658170 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:48.657954 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-kube-api-access-fbt8t\" (UniqueName: \"kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-kube-api-access-fbt8t\") pod \"fbfeab8a-b74e-4188-b565-b56442a01b79\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " Nov 03 14:11:48.658384 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:48.658357 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-registry-tls\" (UniqueName: \"kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-registry-tls\") pod \"fbfeab8a-b74e-4188-b565-b56442a01b79\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " Nov 03 14:11:48.658469 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:48.658377 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbfeab8a-b74e-4188-b565-b56442a01b79-registry-certificates" pod "fbfeab8a-b74e-4188-b565-b56442a01b79" (UID: "fbfeab8a-b74e-4188-b565-b56442a01b79"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:11:48.658469 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:48.658430 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbfeab8a-b74e-4188-b565-b56442a01b79-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fbfeab8a-b74e-4188-b565-b56442a01b79-image-registry-private-configuration\") pod \"fbfeab8a-b74e-4188-b565-b56442a01b79\" (UID: \"fbfeab8a-b74e-4188-b565-b56442a01b79\") " Nov 03 14:11:48.658578 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:48.658550 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbfeab8a-b74e-4188-b565-b56442a01b79-trusted-ca" pod "fbfeab8a-b74e-4188-b565-b56442a01b79" (UID: "fbfeab8a-b74e-4188-b565-b56442a01b79"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:11:48.658774 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:48.658754 5060 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fbfeab8a-b74e-4188-b565-b56442a01b79-registry-certificates\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:11:48.658871 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:48.658779 5060 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fbfeab8a-b74e-4188-b565-b56442a01b79-trusted-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:11:48.660735 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:48.660707 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-kube-api-access-fbt8t" pod "fbfeab8a-b74e-4188-b565-b56442a01b79" (UID: "fbfeab8a-b74e-4188-b565-b56442a01b79"). InnerVolumeSpecName "kube-api-access-fbt8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:11:48.662110 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:48.661191 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-bound-sa-token" pod "fbfeab8a-b74e-4188-b565-b56442a01b79" (UID: "fbfeab8a-b74e-4188-b565-b56442a01b79"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:11:48.662110 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:48.661280 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbfeab8a-b74e-4188-b565-b56442a01b79-image-registry-private-configuration" pod "fbfeab8a-b74e-4188-b565-b56442a01b79" (UID: "fbfeab8a-b74e-4188-b565-b56442a01b79"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:11:48.662110 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:48.661537 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-registry-tls" pod "fbfeab8a-b74e-4188-b565-b56442a01b79" (UID: "fbfeab8a-b74e-4188-b565-b56442a01b79"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:11:48.662278 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:48.662176 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbfeab8a-b74e-4188-b565-b56442a01b79-installation-pull-secrets" pod "fbfeab8a-b74e-4188-b565-b56442a01b79" (UID: "fbfeab8a-b74e-4188-b565-b56442a01b79"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:11:48.667803 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:48.667776 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbfeab8a-b74e-4188-b565-b56442a01b79-ca-trust-extracted" pod "fbfeab8a-b74e-4188-b565-b56442a01b79" (UID: "fbfeab8a-b74e-4188-b565-b56442a01b79"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:11:48.759763 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:48.759728 5060 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fbfeab8a-b74e-4188-b565-b56442a01b79-image-registry-private-configuration\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:11:48.759763 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:48.759762 5060 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fbfeab8a-b74e-4188-b565-b56442a01b79-ca-trust-extracted\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:11:48.759763 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:48.759778 5060 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fbfeab8a-b74e-4188-b565-b56442a01b79-installation-pull-secrets\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:11:48.760040 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:48.759792 5060 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-bound-sa-token\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:11:48.760040 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:48.759804 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbt8t\" (UniqueName: \"kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-kube-api-access-fbt8t\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:11:48.760040 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:48.759817 5060 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fbfeab8a-b74e-4188-b565-b56442a01b79-registry-tls\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:11:49.570637 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:49.570600 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57647df665-9q52v" event={"ID":"fbfeab8a-b74e-4188-b565-b56442a01b79","Type":"ContainerDied","Data":"95901b99c6d073d798f9cf1410ef4a41174bacb7b7ff504c9ede48711d2ff199"} Nov 03 14:11:49.571095 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:49.570659 5060 scope.go:122] "RemoveContainer" containerID="af30d755532f37de5e52d15dc1fb476952fd52dc3c1bcec2b2e7c2700d5a4763" Nov 03 14:11:49.571095 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:49.570616 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57647df665-9q52v" Nov 03 14:11:49.640193 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:49.640167 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57647df665-9q52v"] Nov 03 14:11:49.653060 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:49.653032 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-57647df665-9q52v"] Nov 03 14:11:50.251032 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:50.249538 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbfeab8a-b74e-4188-b565-b56442a01b79" path="/var/lib/kubelet/pods/fbfeab8a-b74e-4188-b565-b56442a01b79/volumes" Nov 03 14:11:55.763053 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:55.763020 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-fzk44-mlg6z"] Nov 03 14:11:55.763589 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:55.763199 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-fzk44-mlg6z" podUID="7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d" containerName="insights-gathering" containerID="cri-o://a380b9bda8356290219aff6192ae489235555e3d14a3bc20f3c5b1e1cbe750a8" gracePeriod=30 Nov 03 14:11:56.017365 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:56.017302 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fzk44-mlg6z" Nov 03 14:11:56.126675 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:56.126631 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d-kube-api-access-m5m26\" (UniqueName: \"kubernetes.io/projected/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d-kube-api-access-m5m26\") pod \"7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d\" (UID: \"7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d\") " Nov 03 14:11:56.126911 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:56.126752 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d-trusted-ca-bundle\") pod \"7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d\" (UID: \"7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d\") " Nov 03 14:11:56.126911 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:56.126776 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d-service-ca-bundle\") pod \"7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d\" (UID: \"7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d\") " Nov 03 14:11:56.127037 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:56.126952 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d-archives-path\") pod \"7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d\" (UID: \"7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d\") " Nov 03 14:11:56.127248 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:56.127212 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d-trusted-ca-bundle" pod "7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d" (UID: "7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:11:56.127248 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:56.127222 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d-service-ca-bundle" pod "7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d" (UID: "7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:11:56.127395 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:56.127257 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d-archives-path" pod "7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d" (UID: "7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:11:56.129085 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:56.129068 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d-kube-api-access-m5m26" pod "7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d" (UID: "7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d"). InnerVolumeSpecName "kube-api-access-m5m26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:11:56.228460 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:56.228423 5060 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d-archives-path\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:11:56.228460 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:56.228457 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5m26\" (UniqueName: \"kubernetes.io/projected/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d-kube-api-access-m5m26\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:11:56.228460 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:56.228469 5060 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d-trusted-ca-bundle\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:11:56.228690 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:56.228478 5060 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d-service-ca-bundle\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:11:56.601003 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:56.600964 5060 generic.go:356] "Generic (PLEG): container finished" podID="7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d" containerID="a380b9bda8356290219aff6192ae489235555e3d14a3bc20f3c5b1e1cbe750a8" exitCode=2 Nov 03 14:11:56.601195 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:56.601025 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fzk44-mlg6z" Nov 03 14:11:56.601195 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:56.601087 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fzk44-mlg6z" event={"ID":"7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d","Type":"ContainerDied","Data":"a380b9bda8356290219aff6192ae489235555e3d14a3bc20f3c5b1e1cbe750a8"} Nov 03 14:11:56.601195 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:56.601148 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fzk44-mlg6z" event={"ID":"7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d","Type":"ContainerDied","Data":"8d03b1024d60e73a8391f33068de1832e67c70d38283aa74f1366514ff8452c5"} Nov 03 14:11:56.601195 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:56.601173 5060 scope.go:122] "RemoveContainer" containerID="a380b9bda8356290219aff6192ae489235555e3d14a3bc20f3c5b1e1cbe750a8" Nov 03 14:11:56.609708 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:56.609690 5060 scope.go:122] "RemoveContainer" containerID="a380b9bda8356290219aff6192ae489235555e3d14a3bc20f3c5b1e1cbe750a8" Nov 03 14:11:56.609982 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:11:56.609961 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a380b9bda8356290219aff6192ae489235555e3d14a3bc20f3c5b1e1cbe750a8\": container with ID starting with a380b9bda8356290219aff6192ae489235555e3d14a3bc20f3c5b1e1cbe750a8 not found: ID does not exist" containerID="a380b9bda8356290219aff6192ae489235555e3d14a3bc20f3c5b1e1cbe750a8" Nov 03 14:11:56.610038 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:56.609990 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a380b9bda8356290219aff6192ae489235555e3d14a3bc20f3c5b1e1cbe750a8"} err="failed to get container status \"a380b9bda8356290219aff6192ae489235555e3d14a3bc20f3c5b1e1cbe750a8\": rpc error: code = NotFound desc = could not find container \"a380b9bda8356290219aff6192ae489235555e3d14a3bc20f3c5b1e1cbe750a8\": container with ID starting with a380b9bda8356290219aff6192ae489235555e3d14a3bc20f3c5b1e1cbe750a8 not found: ID does not exist" Nov 03 14:11:56.655722 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:56.655689 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-fzk44-mlg6z"] Nov 03 14:11:56.668645 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:56.668620 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-fzk44-mlg6z"] Nov 03 14:11:58.246802 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:11:58.246769 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d" path="/var/lib/kubelet/pods/7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d/volumes" Nov 03 14:12:09.437827 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:12:09.437792 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:12:10.574440 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:10.574382 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" podUID="b3260ef2-8ff4-4b70-a3a0-e5386a27d210" containerName="registry" containerID="cri-o://cbdb6cf03b590c5a2a9ce28e2d7babdf7f2a13983310b1564a976a1e873270c9" gracePeriod=30 Nov 03 14:12:10.819888 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:10.819863 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:12:10.957530 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:10.957456 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-ca-trust-extracted\") pod \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " Nov 03 14:12:10.957530 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:10.957507 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-registry-tls\" (UniqueName: \"kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-registry-tls\") pod \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " Nov 03 14:12:10.957726 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:10.957552 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-bound-sa-token\") pod \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " Nov 03 14:12:10.957726 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:10.957583 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-trusted-ca\") pod \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " Nov 03 14:12:10.957726 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:10.957631 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-image-registry-private-configuration\") pod \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " Nov 03 14:12:10.957726 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:10.957650 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-kube-api-access-79cqw\" (UniqueName: \"kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-kube-api-access-79cqw\") pod \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " Nov 03 14:12:10.957726 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:10.957686 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-registry-certificates\") pod \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " Nov 03 14:12:10.957726 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:10.957721 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-installation-pull-secrets\") pod \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\" (UID: \"b3260ef2-8ff4-4b70-a3a0-e5386a27d210\") " Nov 03 14:12:10.958266 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:10.958196 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-registry-certificates" pod "b3260ef2-8ff4-4b70-a3a0-e5386a27d210" (UID: "b3260ef2-8ff4-4b70-a3a0-e5386a27d210"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:12:10.958657 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:10.958625 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-trusted-ca" pod "b3260ef2-8ff4-4b70-a3a0-e5386a27d210" (UID: "b3260ef2-8ff4-4b70-a3a0-e5386a27d210"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:12:10.960493 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:10.960435 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-image-registry-private-configuration" pod "b3260ef2-8ff4-4b70-a3a0-e5386a27d210" (UID: "b3260ef2-8ff4-4b70-a3a0-e5386a27d210"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:12:10.960597 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:10.960498 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-bound-sa-token" pod "b3260ef2-8ff4-4b70-a3a0-e5386a27d210" (UID: "b3260ef2-8ff4-4b70-a3a0-e5386a27d210"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:12:10.960597 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:10.960534 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-registry-tls" pod "b3260ef2-8ff4-4b70-a3a0-e5386a27d210" (UID: "b3260ef2-8ff4-4b70-a3a0-e5386a27d210"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:12:10.960597 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:10.960554 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-installation-pull-secrets" pod "b3260ef2-8ff4-4b70-a3a0-e5386a27d210" (UID: "b3260ef2-8ff4-4b70-a3a0-e5386a27d210"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:12:10.960716 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:10.960605 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-kube-api-access-79cqw" pod "b3260ef2-8ff4-4b70-a3a0-e5386a27d210" (UID: "b3260ef2-8ff4-4b70-a3a0-e5386a27d210"). InnerVolumeSpecName "kube-api-access-79cqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:12:10.967069 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:10.967041 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-ca-trust-extracted" pod "b3260ef2-8ff4-4b70-a3a0-e5386a27d210" (UID: "b3260ef2-8ff4-4b70-a3a0-e5386a27d210"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:12:11.058649 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:11.058610 5060 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-bound-sa-token\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:12:11.058649 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:11.058641 5060 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-trusted-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:12:11.058649 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:11.058653 5060 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-image-registry-private-configuration\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:12:11.058901 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:11.058665 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79cqw\" (UniqueName: \"kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-kube-api-access-79cqw\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:12:11.058901 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:11.058676 5060 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-registry-certificates\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:12:11.058901 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:11.058684 5060 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-installation-pull-secrets\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:12:11.058901 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:11.058693 5060 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-ca-trust-extracted\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:12:11.058901 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:11.058701 5060 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3260ef2-8ff4-4b70-a3a0-e5386a27d210-registry-tls\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:12:11.663897 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:11.663863 5060 generic.go:356] "Generic (PLEG): container finished" podID="b3260ef2-8ff4-4b70-a3a0-e5386a27d210" containerID="cbdb6cf03b590c5a2a9ce28e2d7babdf7f2a13983310b1564a976a1e873270c9" exitCode=0 Nov 03 14:12:11.664334 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:11.663926 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" event={"ID":"b3260ef2-8ff4-4b70-a3a0-e5386a27d210","Type":"ContainerDied","Data":"cbdb6cf03b590c5a2a9ce28e2d7babdf7f2a13983310b1564a976a1e873270c9"} Nov 03 14:12:11.664334 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:11.663964 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" event={"ID":"b3260ef2-8ff4-4b70-a3a0-e5386a27d210","Type":"ContainerDied","Data":"ecb6d105641db003649bba764f30ed8ec95d1fd3e3e0cca57a76f448cdfd294d"} Nov 03 14:12:11.664334 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:11.663963 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d7c596f48-qrcsg" Nov 03 14:12:11.664334 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:11.663978 5060 scope.go:122] "RemoveContainer" containerID="cbdb6cf03b590c5a2a9ce28e2d7babdf7f2a13983310b1564a976a1e873270c9" Nov 03 14:12:11.672684 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:11.672665 5060 scope.go:122] "RemoveContainer" containerID="cbdb6cf03b590c5a2a9ce28e2d7babdf7f2a13983310b1564a976a1e873270c9" Nov 03 14:12:11.672990 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:12:11.672970 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbdb6cf03b590c5a2a9ce28e2d7babdf7f2a13983310b1564a976a1e873270c9\": container with ID starting with cbdb6cf03b590c5a2a9ce28e2d7babdf7f2a13983310b1564a976a1e873270c9 not found: ID does not exist" containerID="cbdb6cf03b590c5a2a9ce28e2d7babdf7f2a13983310b1564a976a1e873270c9" Nov 03 14:12:11.673068 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:11.673012 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbdb6cf03b590c5a2a9ce28e2d7babdf7f2a13983310b1564a976a1e873270c9"} err="failed to get container status \"cbdb6cf03b590c5a2a9ce28e2d7babdf7f2a13983310b1564a976a1e873270c9\": rpc error: code = NotFound desc = could not find container \"cbdb6cf03b590c5a2a9ce28e2d7babdf7f2a13983310b1564a976a1e873270c9\": container with ID starting with cbdb6cf03b590c5a2a9ce28e2d7babdf7f2a13983310b1564a976a1e873270c9 not found: ID does not exist" Nov 03 14:12:11.735068 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:11.735037 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d7c596f48-qrcsg"] Nov 03 14:12:11.747780 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:11.747751 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6d7c596f48-qrcsg"] Nov 03 14:12:12.247570 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:12.247537 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3260ef2-8ff4-4b70-a3a0-e5386a27d210" path="/var/lib/kubelet/pods/b3260ef2-8ff4-4b70-a3a0-e5386a27d210/volumes" Nov 03 14:12:16.687167 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:16.687135 5060 generic.go:356] "Generic (PLEG): container finished" podID="8d74c5e4-f631-4eae-84d5-84749b53ccf8" containerID="8634cdc639b1add0ed6a8e211015302fa0ddc90759373b2fd74a373e9448f40e" exitCode=0 Nov 03 14:12:16.687551 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:16.687202 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nvj5v" event={"ID":"8d74c5e4-f631-4eae-84d5-84749b53ccf8","Type":"ContainerDied","Data":"8634cdc639b1add0ed6a8e211015302fa0ddc90759373b2fd74a373e9448f40e"} Nov 03 14:12:16.687551 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:16.687542 5060 scope.go:122] "RemoveContainer" containerID="8634cdc639b1add0ed6a8e211015302fa0ddc90759373b2fd74a373e9448f40e" Nov 03 14:12:17.693333 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:17.693298 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nvj5v" event={"ID":"8d74c5e4-f631-4eae-84d5-84749b53ccf8","Type":"ContainerStarted","Data":"3b9b1db5db2b190d555a1cb2075a207e8ace260a0aefba932b4c53e68326e35a"} Nov 03 14:12:30.748483 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:30.748445 5060 generic.go:356] "Generic (PLEG): container finished" podID="bf085d57-4ae4-4add-9402-9ee1ce5d795a" containerID="0f629cd99feb4211fca577a43c8e18f37eda996a790d401fb93cddb3f3122c2b" exitCode=0 Nov 03 14:12:30.748966 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:30.748520 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-slvhl" event={"ID":"bf085d57-4ae4-4add-9402-9ee1ce5d795a","Type":"ContainerDied","Data":"0f629cd99feb4211fca577a43c8e18f37eda996a790d401fb93cddb3f3122c2b"} Nov 03 14:12:30.748966 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:30.748911 5060 scope.go:122] "RemoveContainer" containerID="0f629cd99feb4211fca577a43c8e18f37eda996a790d401fb93cddb3f3122c2b" Nov 03 14:12:30.750469 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:30.750447 5060 generic.go:356] "Generic (PLEG): container finished" podID="dd3829b5-04dc-4112-b2d9-087d136ba9ec" containerID="e32c8a3ff556b8c5256d83620697a50e2a76d8147164c28802f6b31e7a750786" exitCode=0 Nov 03 14:12:30.750551 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:30.750530 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-j66qt" event={"ID":"dd3829b5-04dc-4112-b2d9-087d136ba9ec","Type":"ContainerDied","Data":"e32c8a3ff556b8c5256d83620697a50e2a76d8147164c28802f6b31e7a750786"} Nov 03 14:12:30.750608 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:30.750579 5060 scope.go:122] "RemoveContainer" containerID="586cf47171ee95b4f449f8b14c62e58862cde12b2db560a5917d6a54dff60ed1" Nov 03 14:12:30.750873 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:30.750855 5060 scope.go:122] "RemoveContainer" containerID="e32c8a3ff556b8c5256d83620697a50e2a76d8147164c28802f6b31e7a750786" Nov 03 14:12:30.751375 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:12:30.751353 5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"insights-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=insights-operator pod=insights-operator-7c4986d449-j66qt_openshift-insights(dd3829b5-04dc-4112-b2d9-087d136ba9ec)\"" pod="openshift-insights/insights-operator-7c4986d449-j66qt" podUID="dd3829b5-04dc-4112-b2d9-087d136ba9ec" Nov 03 14:12:31.758192 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:31.758155 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-slvhl" event={"ID":"bf085d57-4ae4-4add-9402-9ee1ce5d795a","Type":"ContainerStarted","Data":"6fe6d15d40af4633042e372f6b1c031e0f430d745a3748c9e01190cc9b672b96"} Nov 03 14:12:39.438252 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:12:39.438216 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:12:43.242353 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:43.242321 5060 scope.go:122] "RemoveContainer" containerID="e32c8a3ff556b8c5256d83620697a50e2a76d8147164c28802f6b31e7a750786" Nov 03 14:12:43.781822 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:43.781786 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-mq6lc-lk775"] Nov 03 14:12:43.782041 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:43.782003 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-mq6lc-lk775" podUID="8f6eebaa-adb5-4ba8-b95e-face3189c579" containerName="insights-gathering" containerID="cri-o://1f632f132a553777eefd5caa2880f6dfbdd59aa7d4f3ebf7365968779ddb5844" gracePeriod=30 Nov 03 14:12:43.807130 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:43.807106 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-j66qt" event={"ID":"dd3829b5-04dc-4112-b2d9-087d136ba9ec","Type":"ContainerStarted","Data":"dd3265115fb067febf69d65fdf62adf15a56b43c539edbe69099c7757b7ed18a"} Nov 03 14:12:44.030456 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:44.030431 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-mq6lc-lk775" Nov 03 14:12:44.133431 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:44.133359 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f6eebaa-adb5-4ba8-b95e-face3189c579-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f6eebaa-adb5-4ba8-b95e-face3189c579-trusted-ca-bundle\") pod \"8f6eebaa-adb5-4ba8-b95e-face3189c579\" (UID: \"8f6eebaa-adb5-4ba8-b95e-face3189c579\") " Nov 03 14:12:44.133431 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:44.133424 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f6eebaa-adb5-4ba8-b95e-face3189c579-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8f6eebaa-adb5-4ba8-b95e-face3189c579-archives-path\") pod \"8f6eebaa-adb5-4ba8-b95e-face3189c579\" (UID: \"8f6eebaa-adb5-4ba8-b95e-face3189c579\") " Nov 03 14:12:44.133647 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:44.133450 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f6eebaa-adb5-4ba8-b95e-face3189c579-kube-api-access-nb9d5\" (UniqueName: \"kubernetes.io/projected/8f6eebaa-adb5-4ba8-b95e-face3189c579-kube-api-access-nb9d5\") pod \"8f6eebaa-adb5-4ba8-b95e-face3189c579\" (UID: \"8f6eebaa-adb5-4ba8-b95e-face3189c579\") " Nov 03 14:12:44.133647 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:44.133495 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f6eebaa-adb5-4ba8-b95e-face3189c579-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f6eebaa-adb5-4ba8-b95e-face3189c579-service-ca-bundle\") pod \"8f6eebaa-adb5-4ba8-b95e-face3189c579\" (UID: \"8f6eebaa-adb5-4ba8-b95e-face3189c579\") " Nov 03 14:12:44.133754 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:44.133730 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f6eebaa-adb5-4ba8-b95e-face3189c579-archives-path" pod "8f6eebaa-adb5-4ba8-b95e-face3189c579" (UID: "8f6eebaa-adb5-4ba8-b95e-face3189c579"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:12:44.133864 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:44.133824 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f6eebaa-adb5-4ba8-b95e-face3189c579-trusted-ca-bundle" pod "8f6eebaa-adb5-4ba8-b95e-face3189c579" (UID: "8f6eebaa-adb5-4ba8-b95e-face3189c579"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:12:44.133964 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:44.133945 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f6eebaa-adb5-4ba8-b95e-face3189c579-service-ca-bundle" pod "8f6eebaa-adb5-4ba8-b95e-face3189c579" (UID: "8f6eebaa-adb5-4ba8-b95e-face3189c579"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:12:44.135528 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:44.135508 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f6eebaa-adb5-4ba8-b95e-face3189c579-kube-api-access-nb9d5" pod "8f6eebaa-adb5-4ba8-b95e-face3189c579" (UID: "8f6eebaa-adb5-4ba8-b95e-face3189c579"). InnerVolumeSpecName "kube-api-access-nb9d5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:12:44.234674 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:44.234644 5060 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f6eebaa-adb5-4ba8-b95e-face3189c579-trusted-ca-bundle\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:12:44.234674 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:44.234670 5060 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8f6eebaa-adb5-4ba8-b95e-face3189c579-archives-path\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:12:44.234674 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:44.234680 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb9d5\" (UniqueName: \"kubernetes.io/projected/8f6eebaa-adb5-4ba8-b95e-face3189c579-kube-api-access-nb9d5\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:12:44.234889 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:44.234688 5060 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f6eebaa-adb5-4ba8-b95e-face3189c579-service-ca-bundle\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:12:44.812509 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:44.812476 5060 generic.go:356] "Generic (PLEG): container finished" podID="8f6eebaa-adb5-4ba8-b95e-face3189c579" containerID="1f632f132a553777eefd5caa2880f6dfbdd59aa7d4f3ebf7365968779ddb5844" exitCode=2 Nov 03 14:12:44.813002 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:44.812515 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-mq6lc-lk775" Nov 03 14:12:44.813002 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:44.812528 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-mq6lc-lk775" event={"ID":"8f6eebaa-adb5-4ba8-b95e-face3189c579","Type":"ContainerDied","Data":"1f632f132a553777eefd5caa2880f6dfbdd59aa7d4f3ebf7365968779ddb5844"} Nov 03 14:12:44.813002 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:44.812594 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-mq6lc-lk775" event={"ID":"8f6eebaa-adb5-4ba8-b95e-face3189c579","Type":"ContainerDied","Data":"c628c9690ce6dd066f0d8feacff748142ecd0088225a7bdbc912352e5c89f7d7"} Nov 03 14:12:44.813002 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:44.812628 5060 scope.go:122] "RemoveContainer" containerID="1f632f132a553777eefd5caa2880f6dfbdd59aa7d4f3ebf7365968779ddb5844" Nov 03 14:12:44.820434 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:44.820403 5060 scope.go:122] "RemoveContainer" containerID="1f632f132a553777eefd5caa2880f6dfbdd59aa7d4f3ebf7365968779ddb5844" Nov 03 14:12:44.820707 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:12:44.820687 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f632f132a553777eefd5caa2880f6dfbdd59aa7d4f3ebf7365968779ddb5844\": container with ID starting with 1f632f132a553777eefd5caa2880f6dfbdd59aa7d4f3ebf7365968779ddb5844 not found: ID does not exist" containerID="1f632f132a553777eefd5caa2880f6dfbdd59aa7d4f3ebf7365968779ddb5844" Nov 03 14:12:44.820749 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:44.820718 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f632f132a553777eefd5caa2880f6dfbdd59aa7d4f3ebf7365968779ddb5844"} err="failed to get container status \"1f632f132a553777eefd5caa2880f6dfbdd59aa7d4f3ebf7365968779ddb5844\": rpc error: code = NotFound desc = could not find container \"1f632f132a553777eefd5caa2880f6dfbdd59aa7d4f3ebf7365968779ddb5844\": container with ID starting with 1f632f132a553777eefd5caa2880f6dfbdd59aa7d4f3ebf7365968779ddb5844 not found: ID does not exist" Nov 03 14:12:44.869253 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:44.869219 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-mq6lc-lk775"] Nov 03 14:12:44.881965 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:44.881939 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-mq6lc-lk775"] Nov 03 14:12:46.246746 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:12:46.246702 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f6eebaa-adb5-4ba8-b95e-face3189c579" path="/var/lib/kubelet/pods/8f6eebaa-adb5-4ba8-b95e-face3189c579/volumes" Nov 03 14:13:09.437893 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:13:09.437852 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:13:39.437853 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:13:39.437801 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:14:09.437943 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:14:09.437905 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:14:39.437610 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:14:39.437570 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:15:09.437758 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:15:09.437717 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:15:17.729444 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:17.729406 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bkg69"] Nov 03 14:15:17.729955 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:17.729828 5060 cpu_manager.go:414] "containerMap: removing container" podUID="7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d" containerName="insights-gathering" Nov 03 14:15:17.729955 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:17.729857 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d" containerName="insights-gathering" Nov 03 14:15:17.729955 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:17.729870 5060 cpu_manager.go:414] "containerMap: removing container" podUID="b3260ef2-8ff4-4b70-a3a0-e5386a27d210" containerName="registry" Nov 03 14:15:17.729955 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:17.729875 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3260ef2-8ff4-4b70-a3a0-e5386a27d210" containerName="registry" Nov 03 14:15:17.729955 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:17.729884 5060 cpu_manager.go:414] "containerMap: removing container" podUID="fbfeab8a-b74e-4188-b565-b56442a01b79" containerName="registry" Nov 03 14:15:17.729955 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:17.729889 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbfeab8a-b74e-4188-b565-b56442a01b79" containerName="registry" Nov 03 14:15:17.729955 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:17.729908 5060 cpu_manager.go:414] "containerMap: removing container" podUID="8f6eebaa-adb5-4ba8-b95e-face3189c579" containerName="insights-gathering" Nov 03 14:15:17.729955 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:17.729913 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f6eebaa-adb5-4ba8-b95e-face3189c579" containerName="insights-gathering" Nov 03 14:15:17.730200 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:17.729972 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f6eebaa-adb5-4ba8-b95e-face3189c579" containerName="insights-gathering" Nov 03 14:15:17.730200 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:17.729982 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fe0ba2e-a1f0-41c4-b822-ffdc9c50ae4d" containerName="insights-gathering" Nov 03 14:15:17.730200 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:17.729988 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3260ef2-8ff4-4b70-a3a0-e5386a27d210" containerName="registry" Nov 03 14:15:17.730200 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:17.729993 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbfeab8a-b74e-4188-b565-b56442a01b79" containerName="registry" Nov 03 14:15:17.732973 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:17.732956 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bkg69" Nov 03 14:15:17.762596 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:17.762570 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 14:15:17.762775 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:17.762582 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 14:15:17.762775 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:17.762593 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-p4gq7\"" Nov 03 14:15:17.763712 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:17.763691 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bkg69"] Nov 03 14:15:17.790152 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:17.790128 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9eccb2d0-ae53-4f76-aa75-22c9dd6dee78-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bkg69\" (UID: \"9eccb2d0-ae53-4f76-aa75-22c9dd6dee78\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bkg69" Nov 03 14:15:17.790244 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:17.790163 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9eccb2d0-ae53-4f76-aa75-22c9dd6dee78-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bkg69\" (UID: \"9eccb2d0-ae53-4f76-aa75-22c9dd6dee78\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bkg69" Nov 03 14:15:17.790244 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:17.790192 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvbv8\" (UniqueName: \"kubernetes.io/projected/9eccb2d0-ae53-4f76-aa75-22c9dd6dee78-kube-api-access-dvbv8\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bkg69\" (UID: \"9eccb2d0-ae53-4f76-aa75-22c9dd6dee78\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bkg69" Nov 03 14:15:17.868531 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:17.868499 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w8nl"] Nov 03 14:15:17.871983 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:17.871968 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w8nl" Nov 03 14:15:17.889098 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:17.889079 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-tc4lw\"" Nov 03 14:15:17.891465 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:17.891439 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvbv8\" (UniqueName: \"kubernetes.io/projected/9eccb2d0-ae53-4f76-aa75-22c9dd6dee78-kube-api-access-dvbv8\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bkg69\" (UID: \"9eccb2d0-ae53-4f76-aa75-22c9dd6dee78\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bkg69" Nov 03 14:15:17.891571 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:17.891521 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9eccb2d0-ae53-4f76-aa75-22c9dd6dee78-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bkg69\" (UID: \"9eccb2d0-ae53-4f76-aa75-22c9dd6dee78\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bkg69" Nov 03 14:15:17.891571 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:17.891563 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9eccb2d0-ae53-4f76-aa75-22c9dd6dee78-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bkg69\" (UID: \"9eccb2d0-ae53-4f76-aa75-22c9dd6dee78\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bkg69" Nov 03 14:15:17.891860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:17.891828 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9eccb2d0-ae53-4f76-aa75-22c9dd6dee78-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bkg69\" (UID: \"9eccb2d0-ae53-4f76-aa75-22c9dd6dee78\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bkg69" Nov 03 14:15:17.891907 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:17.891891 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9eccb2d0-ae53-4f76-aa75-22c9dd6dee78-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bkg69\" (UID: \"9eccb2d0-ae53-4f76-aa75-22c9dd6dee78\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bkg69" Nov 03 14:15:17.896441 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:17.896418 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w8nl"] Nov 03 14:15:17.918145 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:17.918116 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvbv8\" (UniqueName: \"kubernetes.io/projected/9eccb2d0-ae53-4f76-aa75-22c9dd6dee78-kube-api-access-dvbv8\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bkg69\" (UID: \"9eccb2d0-ae53-4f76-aa75-22c9dd6dee78\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bkg69" Nov 03 14:15:17.945195 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:17.945128 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvvcx7"] Nov 03 14:15:17.948787 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:17.948765 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvvcx7" Nov 03 14:15:17.973687 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:17.973653 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvvcx7"] Nov 03 14:15:17.989582 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:17.989555 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-92f7l\"" Nov 03 14:15:17.992650 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:17.992625 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e2ef6dd-a993-44b1-91d2-6adba99f539e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w8nl\" (UID: \"0e2ef6dd-a993-44b1-91d2-6adba99f539e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w8nl" Nov 03 14:15:17.992770 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:17.992659 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzppm\" (UniqueName: \"kubernetes.io/projected/0e2ef6dd-a993-44b1-91d2-6adba99f539e-kube-api-access-qzppm\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w8nl\" (UID: \"0e2ef6dd-a993-44b1-91d2-6adba99f539e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w8nl" Nov 03 14:15:17.992852 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:17.992780 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e2ef6dd-a993-44b1-91d2-6adba99f539e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w8nl\" (UID: \"0e2ef6dd-a993-44b1-91d2-6adba99f539e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w8nl" Nov 03 14:15:18.048588 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:18.048560 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bkg69" Nov 03 14:15:18.094366 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:18.093797 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/111e5e75-1f63-4ed0-a195-800984458f04-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvvcx7\" (UID: \"111e5e75-1f63-4ed0-a195-800984458f04\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvvcx7" Nov 03 14:15:18.094366 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:18.093881 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e2ef6dd-a993-44b1-91d2-6adba99f539e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w8nl\" (UID: \"0e2ef6dd-a993-44b1-91d2-6adba99f539e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w8nl" Nov 03 14:15:18.094366 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:18.093981 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzppm\" (UniqueName: \"kubernetes.io/projected/0e2ef6dd-a993-44b1-91d2-6adba99f539e-kube-api-access-qzppm\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w8nl\" (UID: \"0e2ef6dd-a993-44b1-91d2-6adba99f539e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w8nl" Nov 03 14:15:18.094366 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:18.094063 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/111e5e75-1f63-4ed0-a195-800984458f04-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvvcx7\" (UID: \"111e5e75-1f63-4ed0-a195-800984458f04\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvvcx7" Nov 03 14:15:18.094366 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:18.094103 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e2ef6dd-a993-44b1-91d2-6adba99f539e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w8nl\" (UID: \"0e2ef6dd-a993-44b1-91d2-6adba99f539e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w8nl" Nov 03 14:15:18.094366 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:18.094296 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e2ef6dd-a993-44b1-91d2-6adba99f539e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w8nl\" (UID: \"0e2ef6dd-a993-44b1-91d2-6adba99f539e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w8nl" Nov 03 14:15:18.094823 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:18.094799 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e2ef6dd-a993-44b1-91d2-6adba99f539e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w8nl\" (UID: \"0e2ef6dd-a993-44b1-91d2-6adba99f539e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w8nl" Nov 03 14:15:18.096362 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:18.096334 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjwtm\" (UniqueName: \"kubernetes.io/projected/111e5e75-1f63-4ed0-a195-800984458f04-kube-api-access-mjwtm\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvvcx7\" (UID: \"111e5e75-1f63-4ed0-a195-800984458f04\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvvcx7" Nov 03 14:15:18.117582 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:18.117549 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzppm\" (UniqueName: \"kubernetes.io/projected/0e2ef6dd-a993-44b1-91d2-6adba99f539e-kube-api-access-qzppm\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w8nl\" (UID: \"0e2ef6dd-a993-44b1-91d2-6adba99f539e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w8nl" Nov 03 14:15:18.188125 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:18.188092 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w8nl" Nov 03 14:15:18.197455 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:18.197427 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/111e5e75-1f63-4ed0-a195-800984458f04-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvvcx7\" (UID: \"111e5e75-1f63-4ed0-a195-800984458f04\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvvcx7" Nov 03 14:15:18.197527 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:18.197511 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/111e5e75-1f63-4ed0-a195-800984458f04-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvvcx7\" (UID: \"111e5e75-1f63-4ed0-a195-800984458f04\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvvcx7" Nov 03 14:15:18.197577 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:18.197562 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjwtm\" (UniqueName: \"kubernetes.io/projected/111e5e75-1f63-4ed0-a195-800984458f04-kube-api-access-mjwtm\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvvcx7\" (UID: \"111e5e75-1f63-4ed0-a195-800984458f04\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvvcx7" Nov 03 14:15:18.197890 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:18.197867 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/111e5e75-1f63-4ed0-a195-800984458f04-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvvcx7\" (UID: \"111e5e75-1f63-4ed0-a195-800984458f04\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvvcx7" Nov 03 14:15:18.197944 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:18.197875 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/111e5e75-1f63-4ed0-a195-800984458f04-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvvcx7\" (UID: \"111e5e75-1f63-4ed0-a195-800984458f04\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvvcx7" Nov 03 14:15:18.220070 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:18.220042 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjwtm\" (UniqueName: \"kubernetes.io/projected/111e5e75-1f63-4ed0-a195-800984458f04-kube-api-access-mjwtm\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvvcx7\" (UID: \"111e5e75-1f63-4ed0-a195-800984458f04\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvvcx7" Nov 03 14:15:18.264327 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:18.264301 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvvcx7" Nov 03 14:15:18.365205 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:18.365173 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bkg69"] Nov 03 14:15:18.368271 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:15:18.368243 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eccb2d0_ae53_4f76_aa75_22c9dd6dee78.slice/crio-ec1740ca4cec841eb64191c406be82671671ba3fb1388e8b7f6030058a8229fe WatchSource:0}: Error finding container ec1740ca4cec841eb64191c406be82671671ba3fb1388e8b7f6030058a8229fe: Status 404 returned error can't find the container with id ec1740ca4cec841eb64191c406be82671671ba3fb1388e8b7f6030058a8229fe Nov 03 14:15:18.407145 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:18.407116 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bkg69" event={"ID":"9eccb2d0-ae53-4f76-aa75-22c9dd6dee78","Type":"ContainerStarted","Data":"ec1740ca4cec841eb64191c406be82671671ba3fb1388e8b7f6030058a8229fe"} Nov 03 14:15:18.483178 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:18.483101 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w8nl"] Nov 03 14:15:18.485985 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:15:18.485959 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e2ef6dd_a993_44b1_91d2_6adba99f539e.slice/crio-fd69eebc024566454c3856f4dcf54ae0ea4adaac964a88ad6b04479c74857a09 WatchSource:0}: Error finding container fd69eebc024566454c3856f4dcf54ae0ea4adaac964a88ad6b04479c74857a09: Status 404 returned error can't find the container with id fd69eebc024566454c3856f4dcf54ae0ea4adaac964a88ad6b04479c74857a09 Nov 03 14:15:18.564102 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:18.564073 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvvcx7"] Nov 03 14:15:18.567092 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:15:18.567065 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod111e5e75_1f63_4ed0_a195_800984458f04.slice/crio-96ea4de46f7660c8cab515c085401a3dd263f715637ec3680645903eb90c9e1e WatchSource:0}: Error finding container 96ea4de46f7660c8cab515c085401a3dd263f715637ec3680645903eb90c9e1e: Status 404 returned error can't find the container with id 96ea4de46f7660c8cab515c085401a3dd263f715637ec3680645903eb90c9e1e Nov 03 14:15:19.217108 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:19.217069 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czf592"] Nov 03 14:15:19.220980 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:19.220953 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czf592" Nov 03 14:15:19.250347 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:19.250286 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-2zwzh\"" Nov 03 14:15:19.251758 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:19.251733 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czf592"] Nov 03 14:15:19.308313 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:19.308270 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9335b4d1-00f3-4d59-a46c-41b18f06b367-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czf592\" (UID: \"9335b4d1-00f3-4d59-a46c-41b18f06b367\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czf592" Nov 03 14:15:19.308465 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:19.308348 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcwxg\" (UniqueName: \"kubernetes.io/projected/9335b4d1-00f3-4d59-a46c-41b18f06b367-kube-api-access-pcwxg\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czf592\" (UID: \"9335b4d1-00f3-4d59-a46c-41b18f06b367\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czf592" Nov 03 14:15:19.308465 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:19.308377 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9335b4d1-00f3-4d59-a46c-41b18f06b367-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czf592\" (UID: \"9335b4d1-00f3-4d59-a46c-41b18f06b367\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czf592" Nov 03 14:15:19.410209 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:19.410178 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9335b4d1-00f3-4d59-a46c-41b18f06b367-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czf592\" (UID: \"9335b4d1-00f3-4d59-a46c-41b18f06b367\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czf592" Nov 03 14:15:19.410399 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:19.410263 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcwxg\" (UniqueName: \"kubernetes.io/projected/9335b4d1-00f3-4d59-a46c-41b18f06b367-kube-api-access-pcwxg\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czf592\" (UID: \"9335b4d1-00f3-4d59-a46c-41b18f06b367\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czf592" Nov 03 14:15:19.410399 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:19.410294 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9335b4d1-00f3-4d59-a46c-41b18f06b367-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czf592\" (UID: \"9335b4d1-00f3-4d59-a46c-41b18f06b367\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czf592" Nov 03 14:15:19.410707 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:19.410651 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9335b4d1-00f3-4d59-a46c-41b18f06b367-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czf592\" (UID: \"9335b4d1-00f3-4d59-a46c-41b18f06b367\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czf592" Nov 03 14:15:19.411040 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:19.410998 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9335b4d1-00f3-4d59-a46c-41b18f06b367-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czf592\" (UID: \"9335b4d1-00f3-4d59-a46c-41b18f06b367\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czf592" Nov 03 14:15:19.415415 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:19.415377 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvvcx7" event={"ID":"111e5e75-1f63-4ed0-a195-800984458f04","Type":"ContainerStarted","Data":"96ea4de46f7660c8cab515c085401a3dd263f715637ec3680645903eb90c9e1e"} Nov 03 14:15:19.416975 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:19.416947 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w8nl" event={"ID":"0e2ef6dd-a993-44b1-91d2-6adba99f539e","Type":"ContainerStarted","Data":"fd69eebc024566454c3856f4dcf54ae0ea4adaac964a88ad6b04479c74857a09"} Nov 03 14:15:19.451498 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:19.451468 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcwxg\" (UniqueName: \"kubernetes.io/projected/9335b4d1-00f3-4d59-a46c-41b18f06b367-kube-api-access-pcwxg\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czf592\" (UID: \"9335b4d1-00f3-4d59-a46c-41b18f06b367\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czf592" Nov 03 14:15:19.542629 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:19.542592 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czf592" Nov 03 14:15:19.569560 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:19.569230 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ncjkw"] Nov 03 14:15:19.573314 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:19.573125 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ncjkw" Nov 03 14:15:19.592769 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:19.592744 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-sfbzt\"" Nov 03 14:15:19.607725 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:19.601743 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ncjkw"] Nov 03 14:15:19.714469 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:19.714417 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24796a48-abd8-4365-9d68-9fa9f226cf38-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ncjkw\" (UID: \"24796a48-abd8-4365-9d68-9fa9f226cf38\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ncjkw" Nov 03 14:15:19.714641 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:19.714471 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76jbr\" (UniqueName: \"kubernetes.io/projected/24796a48-abd8-4365-9d68-9fa9f226cf38-kube-api-access-76jbr\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ncjkw\" (UID: \"24796a48-abd8-4365-9d68-9fa9f226cf38\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ncjkw" Nov 03 14:15:19.714641 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:19.714514 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24796a48-abd8-4365-9d68-9fa9f226cf38-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ncjkw\" (UID: \"24796a48-abd8-4365-9d68-9fa9f226cf38\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ncjkw" Nov 03 14:15:19.815865 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:19.815762 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24796a48-abd8-4365-9d68-9fa9f226cf38-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ncjkw\" (UID: \"24796a48-abd8-4365-9d68-9fa9f226cf38\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ncjkw" Nov 03 14:15:19.815865 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:19.815814 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76jbr\" (UniqueName: \"kubernetes.io/projected/24796a48-abd8-4365-9d68-9fa9f226cf38-kube-api-access-76jbr\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ncjkw\" (UID: \"24796a48-abd8-4365-9d68-9fa9f226cf38\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ncjkw" Nov 03 14:15:19.816131 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:19.815875 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24796a48-abd8-4365-9d68-9fa9f226cf38-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ncjkw\" (UID: \"24796a48-abd8-4365-9d68-9fa9f226cf38\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ncjkw" Nov 03 14:15:19.816576 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:19.816373 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24796a48-abd8-4365-9d68-9fa9f226cf38-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ncjkw\" (UID: \"24796a48-abd8-4365-9d68-9fa9f226cf38\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ncjkw" Nov 03 14:15:19.816576 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:19.816519 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24796a48-abd8-4365-9d68-9fa9f226cf38-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ncjkw\" (UID: \"24796a48-abd8-4365-9d68-9fa9f226cf38\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ncjkw" Nov 03 14:15:19.842967 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:19.842935 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76jbr\" (UniqueName: \"kubernetes.io/projected/24796a48-abd8-4365-9d68-9fa9f226cf38-kube-api-access-76jbr\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ncjkw\" (UID: \"24796a48-abd8-4365-9d68-9fa9f226cf38\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ncjkw" Nov 03 14:15:19.862288 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:19.862253 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czf592"] Nov 03 14:15:19.868326 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:15:19.868294 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9335b4d1_00f3_4d59_a46c_41b18f06b367.slice/crio-58a0c7059167d12be61b05e50df8acc3f8ad94ecbae19567d038fc41a868663a WatchSource:0}: Error finding container 58a0c7059167d12be61b05e50df8acc3f8ad94ecbae19567d038fc41a868663a: Status 404 returned error can't find the container with id 58a0c7059167d12be61b05e50df8acc3f8ad94ecbae19567d038fc41a868663a Nov 03 14:15:19.892656 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:19.892631 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ncjkw" Nov 03 14:15:20.207973 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:20.207942 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ncjkw"] Nov 03 14:15:20.210363 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:15:20.210336 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24796a48_abd8_4365_9d68_9fa9f226cf38.slice/crio-3efd13fe1d3f87104416b23e92fe482403fa5c3e1580c7fdf2c2cbd79afa39d2 WatchSource:0}: Error finding container 3efd13fe1d3f87104416b23e92fe482403fa5c3e1580c7fdf2c2cbd79afa39d2: Status 404 returned error can't find the container with id 3efd13fe1d3f87104416b23e92fe482403fa5c3e1580c7fdf2c2cbd79afa39d2 Nov 03 14:15:20.427529 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:20.427486 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czf592" event={"ID":"9335b4d1-00f3-4d59-a46c-41b18f06b367","Type":"ContainerStarted","Data":"58a0c7059167d12be61b05e50df8acc3f8ad94ecbae19567d038fc41a868663a"} Nov 03 14:15:20.429294 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:20.429265 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ncjkw" event={"ID":"24796a48-abd8-4365-9d68-9fa9f226cf38","Type":"ContainerStarted","Data":"3efd13fe1d3f87104416b23e92fe482403fa5c3e1580c7fdf2c2cbd79afa39d2"} Nov 03 14:15:28.478007 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:28.477862 5060 generic.go:356] "Generic (PLEG): container finished" podID="0e2ef6dd-a993-44b1-91d2-6adba99f539e" containerID="335bf2892825d6852dd37da832c7dc64d2fe678726a06d8663c7bb4eb07f92dc" exitCode=0 Nov 03 14:15:28.478007 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:28.477955 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w8nl" event={"ID":"0e2ef6dd-a993-44b1-91d2-6adba99f539e","Type":"ContainerDied","Data":"335bf2892825d6852dd37da832c7dc64d2fe678726a06d8663c7bb4eb07f92dc"} Nov 03 14:15:28.480115 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:28.480021 5060 generic.go:356] "Generic (PLEG): container finished" podID="9eccb2d0-ae53-4f76-aa75-22c9dd6dee78" containerID="8abd1a24ca58425ecbfddaf5f091d185749e8f12c2682b40186921413f8d56e8" exitCode=0 Nov 03 14:15:28.480189 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:28.480159 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bkg69" event={"ID":"9eccb2d0-ae53-4f76-aa75-22c9dd6dee78","Type":"ContainerDied","Data":"8abd1a24ca58425ecbfddaf5f091d185749e8f12c2682b40186921413f8d56e8"} Nov 03 14:15:28.482253 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:28.482225 5060 generic.go:356] "Generic (PLEG): container finished" podID="111e5e75-1f63-4ed0-a195-800984458f04" containerID="8fefe554ddb84dd92390560f0fbd9b9496efeadc8347d8c30ee51ce7dadf5572" exitCode=0 Nov 03 14:15:28.482357 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:28.482304 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvvcx7" event={"ID":"111e5e75-1f63-4ed0-a195-800984458f04","Type":"ContainerDied","Data":"8fefe554ddb84dd92390560f0fbd9b9496efeadc8347d8c30ee51ce7dadf5572"} Nov 03 14:15:30.493383 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:30.493344 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bkg69" event={"ID":"9eccb2d0-ae53-4f76-aa75-22c9dd6dee78","Type":"ContainerStarted","Data":"8006047bcec937be970edee2dc6cacd7090d2ea466f7ca340c92894ad0c129b6"} Nov 03 14:15:30.495585 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:30.495555 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czf592" event={"ID":"9335b4d1-00f3-4d59-a46c-41b18f06b367","Type":"ContainerStarted","Data":"6ce2abba47542064acd2c3a329b902022595df868e0c4fd08b4b2bcc860a71f4"} Nov 03 14:15:30.498330 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:30.498297 5060 generic.go:356] "Generic (PLEG): container finished" podID="24796a48-abd8-4365-9d68-9fa9f226cf38" containerID="65e92bf1377c8f2763fb5caf238f8b638589cf9fc1dacff508d97329cad6cc0a" exitCode=0 Nov 03 14:15:30.498434 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:30.498412 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ncjkw" event={"ID":"24796a48-abd8-4365-9d68-9fa9f226cf38","Type":"ContainerDied","Data":"65e92bf1377c8f2763fb5caf238f8b638589cf9fc1dacff508d97329cad6cc0a"} Nov 03 14:15:30.500480 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:30.500455 5060 generic.go:356] "Generic (PLEG): container finished" podID="0e2ef6dd-a993-44b1-91d2-6adba99f539e" containerID="536758c2a6d7ca707ea0696425acd5fc3cb96ccecd0fae35ab7a26e5dcb236c3" exitCode=0 Nov 03 14:15:30.500559 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:30.500499 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w8nl" event={"ID":"0e2ef6dd-a993-44b1-91d2-6adba99f539e","Type":"ContainerDied","Data":"536758c2a6d7ca707ea0696425acd5fc3cb96ccecd0fae35ab7a26e5dcb236c3"} Nov 03 14:15:31.508326 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:31.508271 5060 generic.go:356] "Generic (PLEG): container finished" podID="24796a48-abd8-4365-9d68-9fa9f226cf38" containerID="6309651e1c85e3b950b001d2a657feb847f05667b9dbb1923170afe803a45841" exitCode=0 Nov 03 14:15:31.508862 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:31.508323 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ncjkw" event={"ID":"24796a48-abd8-4365-9d68-9fa9f226cf38","Type":"ContainerDied","Data":"6309651e1c85e3b950b001d2a657feb847f05667b9dbb1923170afe803a45841"} Nov 03 14:15:31.511215 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:31.511159 5060 generic.go:356] "Generic (PLEG): container finished" podID="9eccb2d0-ae53-4f76-aa75-22c9dd6dee78" containerID="8006047bcec937be970edee2dc6cacd7090d2ea466f7ca340c92894ad0c129b6" exitCode=0 Nov 03 14:15:31.511292 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:31.511276 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bkg69" event={"ID":"9eccb2d0-ae53-4f76-aa75-22c9dd6dee78","Type":"ContainerDied","Data":"8006047bcec937be970edee2dc6cacd7090d2ea466f7ca340c92894ad0c129b6"} Nov 03 14:15:31.513156 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:31.513129 5060 generic.go:356] "Generic (PLEG): container finished" podID="9335b4d1-00f3-4d59-a46c-41b18f06b367" containerID="6ce2abba47542064acd2c3a329b902022595df868e0c4fd08b4b2bcc860a71f4" exitCode=0 Nov 03 14:15:31.513261 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:31.513182 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czf592" event={"ID":"9335b4d1-00f3-4d59-a46c-41b18f06b367","Type":"ContainerDied","Data":"6ce2abba47542064acd2c3a329b902022595df868e0c4fd08b4b2bcc860a71f4"} Nov 03 14:15:38.559498 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:38.559458 5060 generic.go:356] "Generic (PLEG): container finished" podID="0e2ef6dd-a993-44b1-91d2-6adba99f539e" containerID="38f3f12032c6d2e75c175d53e9136478f6f0e445ca26754027cc4d7a7372b9c7" exitCode=0 Nov 03 14:15:38.559946 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:38.559629 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w8nl" event={"ID":"0e2ef6dd-a993-44b1-91d2-6adba99f539e","Type":"ContainerDied","Data":"38f3f12032c6d2e75c175d53e9136478f6f0e445ca26754027cc4d7a7372b9c7"} Nov 03 14:15:38.562293 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:38.562178 5060 generic.go:356] "Generic (PLEG): container finished" podID="111e5e75-1f63-4ed0-a195-800984458f04" containerID="2759a7e5c50875807452327b4140bbc6c0371e765e1e0b4974b0275095a41432" exitCode=0 Nov 03 14:15:38.562293 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:38.562252 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvvcx7" event={"ID":"111e5e75-1f63-4ed0-a195-800984458f04","Type":"ContainerDied","Data":"2759a7e5c50875807452327b4140bbc6c0371e765e1e0b4974b0275095a41432"} Nov 03 14:15:39.447182 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:15:39.447149 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:15:39.568711 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:39.568676 5060 generic.go:356] "Generic (PLEG): container finished" podID="9335b4d1-00f3-4d59-a46c-41b18f06b367" containerID="50039fdd8e13d1ce71fb56c3fc5101aee80b04df3db1fb3a5469ff6b38c5bd40" exitCode=0 Nov 03 14:15:39.569202 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:39.568733 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czf592" event={"ID":"9335b4d1-00f3-4d59-a46c-41b18f06b367","Type":"ContainerDied","Data":"50039fdd8e13d1ce71fb56c3fc5101aee80b04df3db1fb3a5469ff6b38c5bd40"} Nov 03 14:15:39.571279 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:39.571238 5060 generic.go:356] "Generic (PLEG): container finished" podID="111e5e75-1f63-4ed0-a195-800984458f04" containerID="52808abc43f215de3a4667845914654b3d2ed61723e7561162a132a5381fd25b" exitCode=0 Nov 03 14:15:39.571412 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:39.571291 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvvcx7" event={"ID":"111e5e75-1f63-4ed0-a195-800984458f04","Type":"ContainerDied","Data":"52808abc43f215de3a4667845914654b3d2ed61723e7561162a132a5381fd25b"} Nov 03 14:15:39.573967 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:39.573942 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ncjkw" event={"ID":"24796a48-abd8-4365-9d68-9fa9f226cf38","Type":"ContainerStarted","Data":"d40df36ae026ba2cad02452f10a972af7f811679364fe6ed57f15dc50ec30f02"} Nov 03 14:15:39.577489 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:39.577470 5060 generic.go:356] "Generic (PLEG): container finished" podID="9eccb2d0-ae53-4f76-aa75-22c9dd6dee78" containerID="8bc44896ff1315c5627fc55205e678e8fbce3a533607323c50f6c2dfc0595b1a" exitCode=0 Nov 03 14:15:39.577759 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:39.577741 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bkg69" event={"ID":"9eccb2d0-ae53-4f76-aa75-22c9dd6dee78","Type":"ContainerDied","Data":"8bc44896ff1315c5627fc55205e678e8fbce3a533607323c50f6c2dfc0595b1a"} Nov 03 14:15:39.668922 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:39.668865 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ncjkw" podStartSLOduration=2.3800804749999998 podStartE2EDuration="20.668846836s" totalImagesPullingTime="18.288766361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:15:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:15:39.664693151 +0000 UTC m=+273.553482735" watchObservedRunningTime="2026-11-03 14:15:39.668846836 +0000 UTC m=+273.557636408" Nov 03 14:15:39.725987 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:39.725955 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w8nl" Nov 03 14:15:39.807711 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:39.807632 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e2ef6dd-a993-44b1-91d2-6adba99f539e-util\" (UniqueName: \"kubernetes.io/empty-dir/0e2ef6dd-a993-44b1-91d2-6adba99f539e-util\") pod \"0e2ef6dd-a993-44b1-91d2-6adba99f539e\" (UID: \"0e2ef6dd-a993-44b1-91d2-6adba99f539e\") " Nov 03 14:15:39.807852 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:39.807715 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e2ef6dd-a993-44b1-91d2-6adba99f539e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e2ef6dd-a993-44b1-91d2-6adba99f539e-bundle\") pod \"0e2ef6dd-a993-44b1-91d2-6adba99f539e\" (UID: \"0e2ef6dd-a993-44b1-91d2-6adba99f539e\") " Nov 03 14:15:39.807852 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:39.807794 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e2ef6dd-a993-44b1-91d2-6adba99f539e-kube-api-access-qzppm\" (UniqueName: \"kubernetes.io/projected/0e2ef6dd-a993-44b1-91d2-6adba99f539e-kube-api-access-qzppm\") pod \"0e2ef6dd-a993-44b1-91d2-6adba99f539e\" (UID: \"0e2ef6dd-a993-44b1-91d2-6adba99f539e\") " Nov 03 14:15:39.808254 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:39.808222 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e2ef6dd-a993-44b1-91d2-6adba99f539e-bundle" pod "0e2ef6dd-a993-44b1-91d2-6adba99f539e" (UID: "0e2ef6dd-a993-44b1-91d2-6adba99f539e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:15:39.810544 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:39.810512 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e2ef6dd-a993-44b1-91d2-6adba99f539e-kube-api-access-qzppm" pod "0e2ef6dd-a993-44b1-91d2-6adba99f539e" (UID: "0e2ef6dd-a993-44b1-91d2-6adba99f539e"). InnerVolumeSpecName "kube-api-access-qzppm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:15:39.812008 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:39.811985 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e2ef6dd-a993-44b1-91d2-6adba99f539e-util" pod "0e2ef6dd-a993-44b1-91d2-6adba99f539e" (UID: "0e2ef6dd-a993-44b1-91d2-6adba99f539e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:15:39.909051 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:39.909021 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzppm\" (UniqueName: \"kubernetes.io/projected/0e2ef6dd-a993-44b1-91d2-6adba99f539e-kube-api-access-qzppm\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:15:39.909051 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:39.909051 5060 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e2ef6dd-a993-44b1-91d2-6adba99f539e-util\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:15:39.909186 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:39.909062 5060 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e2ef6dd-a993-44b1-91d2-6adba99f539e-bundle\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:15:40.583788 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:40.583709 5060 generic.go:356] "Generic (PLEG): container finished" podID="24796a48-abd8-4365-9d68-9fa9f226cf38" containerID="d40df36ae026ba2cad02452f10a972af7f811679364fe6ed57f15dc50ec30f02" exitCode=0 Nov 03 14:15:40.584225 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:40.583798 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ncjkw" event={"ID":"24796a48-abd8-4365-9d68-9fa9f226cf38","Type":"ContainerDied","Data":"d40df36ae026ba2cad02452f10a972af7f811679364fe6ed57f15dc50ec30f02"} Nov 03 14:15:40.585451 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:40.585423 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w8nl" event={"ID":"0e2ef6dd-a993-44b1-91d2-6adba99f539e","Type":"ContainerDied","Data":"fd69eebc024566454c3856f4dcf54ae0ea4adaac964a88ad6b04479c74857a09"} Nov 03 14:15:40.585567 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:40.585470 5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd69eebc024566454c3856f4dcf54ae0ea4adaac964a88ad6b04479c74857a09" Nov 03 14:15:40.585567 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:40.585437 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w8nl" Nov 03 14:15:40.587667 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:40.587643 5060 generic.go:356] "Generic (PLEG): container finished" podID="9335b4d1-00f3-4d59-a46c-41b18f06b367" containerID="3c91f0acf719f44c51af4df2208b6aedddecc12dbe9b8b0df4196d4701e3f7a0" exitCode=0 Nov 03 14:15:40.587781 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:40.587727 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czf592" event={"ID":"9335b4d1-00f3-4d59-a46c-41b18f06b367","Type":"ContainerDied","Data":"3c91f0acf719f44c51af4df2208b6aedddecc12dbe9b8b0df4196d4701e3f7a0"} Nov 03 14:15:40.726043 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:40.726016 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bkg69" Nov 03 14:15:40.756384 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:40.756361 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvvcx7" Nov 03 14:15:40.815691 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:40.815658 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/111e5e75-1f63-4ed0-a195-800984458f04-bundle\" (UniqueName: \"kubernetes.io/empty-dir/111e5e75-1f63-4ed0-a195-800984458f04-bundle\") pod \"111e5e75-1f63-4ed0-a195-800984458f04\" (UID: \"111e5e75-1f63-4ed0-a195-800984458f04\") " Nov 03 14:15:40.815923 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:40.815701 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/111e5e75-1f63-4ed0-a195-800984458f04-util\" (UniqueName: \"kubernetes.io/empty-dir/111e5e75-1f63-4ed0-a195-800984458f04-util\") pod \"111e5e75-1f63-4ed0-a195-800984458f04\" (UID: \"111e5e75-1f63-4ed0-a195-800984458f04\") " Nov 03 14:15:40.815923 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:40.815749 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eccb2d0-ae53-4f76-aa75-22c9dd6dee78-kube-api-access-dvbv8\" (UniqueName: \"kubernetes.io/projected/9eccb2d0-ae53-4f76-aa75-22c9dd6dee78-kube-api-access-dvbv8\") pod \"9eccb2d0-ae53-4f76-aa75-22c9dd6dee78\" (UID: \"9eccb2d0-ae53-4f76-aa75-22c9dd6dee78\") " Nov 03 14:15:40.815923 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:40.815782 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eccb2d0-ae53-4f76-aa75-22c9dd6dee78-util\" (UniqueName: \"kubernetes.io/empty-dir/9eccb2d0-ae53-4f76-aa75-22c9dd6dee78-util\") pod \"9eccb2d0-ae53-4f76-aa75-22c9dd6dee78\" (UID: \"9eccb2d0-ae53-4f76-aa75-22c9dd6dee78\") " Nov 03 14:15:40.815923 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:40.815802 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/111e5e75-1f63-4ed0-a195-800984458f04-kube-api-access-mjwtm\" (UniqueName: \"kubernetes.io/projected/111e5e75-1f63-4ed0-a195-800984458f04-kube-api-access-mjwtm\") pod \"111e5e75-1f63-4ed0-a195-800984458f04\" (UID: \"111e5e75-1f63-4ed0-a195-800984458f04\") " Nov 03 14:15:40.815923 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:40.815856 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eccb2d0-ae53-4f76-aa75-22c9dd6dee78-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9eccb2d0-ae53-4f76-aa75-22c9dd6dee78-bundle\") pod \"9eccb2d0-ae53-4f76-aa75-22c9dd6dee78\" (UID: \"9eccb2d0-ae53-4f76-aa75-22c9dd6dee78\") " Nov 03 14:15:40.816435 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:40.816402 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/111e5e75-1f63-4ed0-a195-800984458f04-bundle" pod "111e5e75-1f63-4ed0-a195-800984458f04" (UID: "111e5e75-1f63-4ed0-a195-800984458f04"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:15:40.816541 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:40.816494 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eccb2d0-ae53-4f76-aa75-22c9dd6dee78-bundle" pod "9eccb2d0-ae53-4f76-aa75-22c9dd6dee78" (UID: "9eccb2d0-ae53-4f76-aa75-22c9dd6dee78"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:15:40.818558 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:40.818528 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/111e5e75-1f63-4ed0-a195-800984458f04-kube-api-access-mjwtm" pod "111e5e75-1f63-4ed0-a195-800984458f04" (UID: "111e5e75-1f63-4ed0-a195-800984458f04"). InnerVolumeSpecName "kube-api-access-mjwtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:15:40.818679 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:40.818660 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eccb2d0-ae53-4f76-aa75-22c9dd6dee78-kube-api-access-dvbv8" pod "9eccb2d0-ae53-4f76-aa75-22c9dd6dee78" (UID: "9eccb2d0-ae53-4f76-aa75-22c9dd6dee78"). InnerVolumeSpecName "kube-api-access-dvbv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:15:40.822893 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:40.822869 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eccb2d0-ae53-4f76-aa75-22c9dd6dee78-util" pod "9eccb2d0-ae53-4f76-aa75-22c9dd6dee78" (UID: "9eccb2d0-ae53-4f76-aa75-22c9dd6dee78"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:15:40.823038 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:40.823017 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/111e5e75-1f63-4ed0-a195-800984458f04-util" pod "111e5e75-1f63-4ed0-a195-800984458f04" (UID: "111e5e75-1f63-4ed0-a195-800984458f04"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:15:40.916738 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:40.916656 5060 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/111e5e75-1f63-4ed0-a195-800984458f04-bundle\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:15:40.916738 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:40.916685 5060 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/111e5e75-1f63-4ed0-a195-800984458f04-util\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:15:40.916738 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:40.916694 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvbv8\" (UniqueName: \"kubernetes.io/projected/9eccb2d0-ae53-4f76-aa75-22c9dd6dee78-kube-api-access-dvbv8\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:15:40.916738 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:40.916703 5060 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9eccb2d0-ae53-4f76-aa75-22c9dd6dee78-util\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:15:40.916738 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:40.916715 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjwtm\" (UniqueName: \"kubernetes.io/projected/111e5e75-1f63-4ed0-a195-800984458f04-kube-api-access-mjwtm\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:15:40.916738 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:40.916723 5060 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9eccb2d0-ae53-4f76-aa75-22c9dd6dee78-bundle\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:15:41.593716 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:41.593678 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bkg69" Nov 03 14:15:41.594198 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:41.593688 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bkg69" event={"ID":"9eccb2d0-ae53-4f76-aa75-22c9dd6dee78","Type":"ContainerDied","Data":"ec1740ca4cec841eb64191c406be82671671ba3fb1388e8b7f6030058a8229fe"} Nov 03 14:15:41.594198 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:41.593818 5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec1740ca4cec841eb64191c406be82671671ba3fb1388e8b7f6030058a8229fe" Nov 03 14:15:41.595714 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:41.595692 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvvcx7" Nov 03 14:15:41.595862 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:41.595723 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvvcx7" event={"ID":"111e5e75-1f63-4ed0-a195-800984458f04","Type":"ContainerDied","Data":"96ea4de46f7660c8cab515c085401a3dd263f715637ec3680645903eb90c9e1e"} Nov 03 14:15:41.595862 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:41.595763 5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96ea4de46f7660c8cab515c085401a3dd263f715637ec3680645903eb90c9e1e" Nov 03 14:15:41.720575 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:41.720552 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czf592" Nov 03 14:15:41.722917 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:41.722900 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9335b4d1-00f3-4d59-a46c-41b18f06b367-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9335b4d1-00f3-4d59-a46c-41b18f06b367-bundle\") pod \"9335b4d1-00f3-4d59-a46c-41b18f06b367\" (UID: \"9335b4d1-00f3-4d59-a46c-41b18f06b367\") " Nov 03 14:15:41.722991 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:41.722978 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9335b4d1-00f3-4d59-a46c-41b18f06b367-util\" (UniqueName: \"kubernetes.io/empty-dir/9335b4d1-00f3-4d59-a46c-41b18f06b367-util\") pod \"9335b4d1-00f3-4d59-a46c-41b18f06b367\" (UID: \"9335b4d1-00f3-4d59-a46c-41b18f06b367\") " Nov 03 14:15:41.723038 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:41.723022 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9335b4d1-00f3-4d59-a46c-41b18f06b367-kube-api-access-pcwxg\" (UniqueName: \"kubernetes.io/projected/9335b4d1-00f3-4d59-a46c-41b18f06b367-kube-api-access-pcwxg\") pod \"9335b4d1-00f3-4d59-a46c-41b18f06b367\" (UID: \"9335b4d1-00f3-4d59-a46c-41b18f06b367\") " Nov 03 14:15:41.723678 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:41.723652 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9335b4d1-00f3-4d59-a46c-41b18f06b367-bundle" pod "9335b4d1-00f3-4d59-a46c-41b18f06b367" (UID: "9335b4d1-00f3-4d59-a46c-41b18f06b367"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:15:41.726315 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:41.726290 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9335b4d1-00f3-4d59-a46c-41b18f06b367-kube-api-access-pcwxg" pod "9335b4d1-00f3-4d59-a46c-41b18f06b367" (UID: "9335b4d1-00f3-4d59-a46c-41b18f06b367"). InnerVolumeSpecName "kube-api-access-pcwxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:15:41.728231 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:41.728202 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9335b4d1-00f3-4d59-a46c-41b18f06b367-util" pod "9335b4d1-00f3-4d59-a46c-41b18f06b367" (UID: "9335b4d1-00f3-4d59-a46c-41b18f06b367"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:15:41.757885 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:41.757864 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ncjkw" Nov 03 14:15:41.823814 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:41.823786 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24796a48-abd8-4365-9d68-9fa9f226cf38-kube-api-access-76jbr\" (UniqueName: \"kubernetes.io/projected/24796a48-abd8-4365-9d68-9fa9f226cf38-kube-api-access-76jbr\") pod \"24796a48-abd8-4365-9d68-9fa9f226cf38\" (UID: \"24796a48-abd8-4365-9d68-9fa9f226cf38\") " Nov 03 14:15:41.823987 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:41.823851 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24796a48-abd8-4365-9d68-9fa9f226cf38-util\" (UniqueName: \"kubernetes.io/empty-dir/24796a48-abd8-4365-9d68-9fa9f226cf38-util\") pod \"24796a48-abd8-4365-9d68-9fa9f226cf38\" (UID: \"24796a48-abd8-4365-9d68-9fa9f226cf38\") " Nov 03 14:15:41.823987 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:41.823960 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24796a48-abd8-4365-9d68-9fa9f226cf38-bundle\" (UniqueName: \"kubernetes.io/empty-dir/24796a48-abd8-4365-9d68-9fa9f226cf38-bundle\") pod \"24796a48-abd8-4365-9d68-9fa9f226cf38\" (UID: \"24796a48-abd8-4365-9d68-9fa9f226cf38\") " Nov 03 14:15:41.824161 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:41.824147 5060 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9335b4d1-00f3-4d59-a46c-41b18f06b367-util\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:15:41.824214 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:41.824169 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcwxg\" (UniqueName: \"kubernetes.io/projected/9335b4d1-00f3-4d59-a46c-41b18f06b367-kube-api-access-pcwxg\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:15:41.824214 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:41.824184 5060 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9335b4d1-00f3-4d59-a46c-41b18f06b367-bundle\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:15:41.824550 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:41.824524 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24796a48-abd8-4365-9d68-9fa9f226cf38-bundle" pod "24796a48-abd8-4365-9d68-9fa9f226cf38" (UID: "24796a48-abd8-4365-9d68-9fa9f226cf38"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:15:41.825797 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:41.825775 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24796a48-abd8-4365-9d68-9fa9f226cf38-kube-api-access-76jbr" pod "24796a48-abd8-4365-9d68-9fa9f226cf38" (UID: "24796a48-abd8-4365-9d68-9fa9f226cf38"). InnerVolumeSpecName "kube-api-access-76jbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:15:41.828518 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:41.828492 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24796a48-abd8-4365-9d68-9fa9f226cf38-util" pod "24796a48-abd8-4365-9d68-9fa9f226cf38" (UID: "24796a48-abd8-4365-9d68-9fa9f226cf38"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:15:41.924925 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:41.924820 5060 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24796a48-abd8-4365-9d68-9fa9f226cf38-bundle\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:15:41.924925 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:41.924875 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76jbr\" (UniqueName: \"kubernetes.io/projected/24796a48-abd8-4365-9d68-9fa9f226cf38-kube-api-access-76jbr\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:15:41.924925 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:41.924888 5060 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24796a48-abd8-4365-9d68-9fa9f226cf38-util\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:15:42.601586 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:42.601501 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czf592" Nov 03 14:15:42.601586 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:42.601532 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czf592" event={"ID":"9335b4d1-00f3-4d59-a46c-41b18f06b367","Type":"ContainerDied","Data":"58a0c7059167d12be61b05e50df8acc3f8ad94ecbae19567d038fc41a868663a"} Nov 03 14:15:42.601586 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:42.601576 5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58a0c7059167d12be61b05e50df8acc3f8ad94ecbae19567d038fc41a868663a" Nov 03 14:15:42.603814 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:42.603784 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ncjkw" event={"ID":"24796a48-abd8-4365-9d68-9fa9f226cf38","Type":"ContainerDied","Data":"3efd13fe1d3f87104416b23e92fe482403fa5c3e1580c7fdf2c2cbd79afa39d2"} Nov 03 14:15:42.603957 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:42.603830 5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3efd13fe1d3f87104416b23e92fe482403fa5c3e1580c7fdf2c2cbd79afa39d2" Nov 03 14:15:42.603957 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:42.603796 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ncjkw" Nov 03 14:15:54.020016 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.019977 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-t7s86"] Nov 03 14:15:54.020403 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.020318 5060 cpu_manager.go:414] "containerMap: removing container" podUID="0e2ef6dd-a993-44b1-91d2-6adba99f539e" containerName="extract" Nov 03 14:15:54.020403 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.020329 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e2ef6dd-a993-44b1-91d2-6adba99f539e" containerName="extract" Nov 03 14:15:54.020403 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.020340 5060 cpu_manager.go:414] "containerMap: removing container" podUID="111e5e75-1f63-4ed0-a195-800984458f04" containerName="extract" Nov 03 14:15:54.020403 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.020345 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="111e5e75-1f63-4ed0-a195-800984458f04" containerName="extract" Nov 03 14:15:54.020403 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.020353 5060 cpu_manager.go:414] "containerMap: removing container" podUID="9335b4d1-00f3-4d59-a46c-41b18f06b367" containerName="pull" Nov 03 14:15:54.020403 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.020358 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9335b4d1-00f3-4d59-a46c-41b18f06b367" containerName="pull" Nov 03 14:15:54.020403 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.020375 5060 cpu_manager.go:414] "containerMap: removing container" podUID="9eccb2d0-ae53-4f76-aa75-22c9dd6dee78" containerName="extract" Nov 03 14:15:54.020403 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.020380 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eccb2d0-ae53-4f76-aa75-22c9dd6dee78" containerName="extract" Nov 03 14:15:54.020403 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.020388 5060 cpu_manager.go:414] "containerMap: removing container" podUID="24796a48-abd8-4365-9d68-9fa9f226cf38" containerName="extract" Nov 03 14:15:54.020403 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.020393 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24796a48-abd8-4365-9d68-9fa9f226cf38" containerName="extract" Nov 03 14:15:54.020403 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.020402 5060 cpu_manager.go:414] "containerMap: removing container" podUID="9335b4d1-00f3-4d59-a46c-41b18f06b367" containerName="extract" Nov 03 14:15:54.020403 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.020407 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9335b4d1-00f3-4d59-a46c-41b18f06b367" containerName="extract" Nov 03 14:15:54.020759 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.020414 5060 cpu_manager.go:414] "containerMap: removing container" podUID="24796a48-abd8-4365-9d68-9fa9f226cf38" containerName="pull" Nov 03 14:15:54.020759 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.020418 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24796a48-abd8-4365-9d68-9fa9f226cf38" containerName="pull" Nov 03 14:15:54.020759 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.020425 5060 cpu_manager.go:414] "containerMap: removing container" podUID="9eccb2d0-ae53-4f76-aa75-22c9dd6dee78" containerName="util" Nov 03 14:15:54.020759 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.020430 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eccb2d0-ae53-4f76-aa75-22c9dd6dee78" containerName="util" Nov 03 14:15:54.020759 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.020440 5060 cpu_manager.go:414] "containerMap: removing container" podUID="0e2ef6dd-a993-44b1-91d2-6adba99f539e" containerName="pull" Nov 03 14:15:54.020759 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.020444 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e2ef6dd-a993-44b1-91d2-6adba99f539e" containerName="pull" Nov 03 14:15:54.020759 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.020451 5060 cpu_manager.go:414] "containerMap: removing container" podUID="9eccb2d0-ae53-4f76-aa75-22c9dd6dee78" containerName="pull" Nov 03 14:15:54.020759 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.020455 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eccb2d0-ae53-4f76-aa75-22c9dd6dee78" containerName="pull" Nov 03 14:15:54.020759 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.020463 5060 cpu_manager.go:414] "containerMap: removing container" podUID="24796a48-abd8-4365-9d68-9fa9f226cf38" containerName="util" Nov 03 14:15:54.020759 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.020467 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24796a48-abd8-4365-9d68-9fa9f226cf38" containerName="util" Nov 03 14:15:54.020759 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.020473 5060 cpu_manager.go:414] "containerMap: removing container" podUID="111e5e75-1f63-4ed0-a195-800984458f04" containerName="pull" Nov 03 14:15:54.020759 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.020477 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="111e5e75-1f63-4ed0-a195-800984458f04" containerName="pull" Nov 03 14:15:54.020759 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.020484 5060 cpu_manager.go:414] "containerMap: removing container" podUID="0e2ef6dd-a993-44b1-91d2-6adba99f539e" containerName="util" Nov 03 14:15:54.020759 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.020488 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e2ef6dd-a993-44b1-91d2-6adba99f539e" containerName="util" Nov 03 14:15:54.020759 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.020494 5060 cpu_manager.go:414] "containerMap: removing container" podUID="111e5e75-1f63-4ed0-a195-800984458f04" containerName="util" Nov 03 14:15:54.020759 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.020498 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="111e5e75-1f63-4ed0-a195-800984458f04" containerName="util" Nov 03 14:15:54.020759 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.020503 5060 cpu_manager.go:414] "containerMap: removing container" podUID="9335b4d1-00f3-4d59-a46c-41b18f06b367" containerName="util" Nov 03 14:15:54.020759 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.020507 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9335b4d1-00f3-4d59-a46c-41b18f06b367" containerName="util" Nov 03 14:15:54.020759 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.020552 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="9335b4d1-00f3-4d59-a46c-41b18f06b367" containerName="extract" Nov 03 14:15:54.020759 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.020564 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e2ef6dd-a993-44b1-91d2-6adba99f539e" containerName="extract" Nov 03 14:15:54.020759 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.020569 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="111e5e75-1f63-4ed0-a195-800984458f04" containerName="extract" Nov 03 14:15:54.020759 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.020576 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eccb2d0-ae53-4f76-aa75-22c9dd6dee78" containerName="extract" Nov 03 14:15:54.020759 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.020581 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="24796a48-abd8-4365-9d68-9fa9f226cf38" containerName="extract" Nov 03 14:15:54.023469 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.023452 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-t7s86" Nov 03 14:15:54.059940 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.059911 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 03 14:15:54.059940 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.059912 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 14:15:54.060113 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.059911 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 14:15:54.060113 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.059939 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-q7cfq\"" Nov 03 14:15:54.061729 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.061706 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-t7s86"] Nov 03 14:15:54.118545 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.118515 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nv2s\" (UniqueName: \"kubernetes.io/projected/57e263ee-fe23-475e-aa0d-72f40f837809-kube-api-access-8nv2s\") pod \"dns-operator-controller-manager-8557cbdb84-t7s86\" (UID: \"57e263ee-fe23-475e-aa0d-72f40f837809\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-t7s86" Nov 03 14:15:54.219389 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.219358 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nv2s\" (UniqueName: \"kubernetes.io/projected/57e263ee-fe23-475e-aa0d-72f40f837809-kube-api-access-8nv2s\") pod \"dns-operator-controller-manager-8557cbdb84-t7s86\" (UID: \"57e263ee-fe23-475e-aa0d-72f40f837809\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-t7s86" Nov 03 14:15:54.250378 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.250346 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nv2s\" (UniqueName: \"kubernetes.io/projected/57e263ee-fe23-475e-aa0d-72f40f837809-kube-api-access-8nv2s\") pod \"dns-operator-controller-manager-8557cbdb84-t7s86\" (UID: \"57e263ee-fe23-475e-aa0d-72f40f837809\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-t7s86" Nov 03 14:15:54.336582 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.336507 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-t7s86" Nov 03 14:15:54.644047 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.643965 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-t7s86"] Nov 03 14:15:54.647710 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:15:54.647680 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57e263ee_fe23_475e_aa0d_72f40f837809.slice/crio-eff86e4597c0bedb329a911b76e8ca6ffa041401847c8bb099b96a73cf30c37e WatchSource:0}: Error finding container eff86e4597c0bedb329a911b76e8ca6ffa041401847c8bb099b96a73cf30c37e: Status 404 returned error can't find the container with id eff86e4597c0bedb329a911b76e8ca6ffa041401847c8bb099b96a73cf30c37e Nov 03 14:15:54.664331 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:54.663019 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-t7s86" event={"ID":"57e263ee-fe23-475e-aa0d-72f40f837809","Type":"ContainerStarted","Data":"eff86e4597c0bedb329a911b76e8ca6ffa041401847c8bb099b96a73cf30c37e"} Nov 03 14:15:56.520525 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:56.520492 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kw7tc"] Nov 03 14:15:56.528699 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:56.528674 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kw7tc" Nov 03 14:15:56.545074 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:56.545050 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kw7tc"] Nov 03 14:15:56.555665 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:56.555626 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 03 14:15:56.555665 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:56.555626 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 03 14:15:56.555856 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:56.555741 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-v8xst\"" Nov 03 14:15:56.641113 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:56.641080 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p86v\" (UniqueName: \"kubernetes.io/projected/31bff3ab-968d-4304-a5fb-3da783dc00f0-kube-api-access-4p86v\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-kw7tc\" (UID: \"31bff3ab-968d-4304-a5fb-3da783dc00f0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kw7tc" Nov 03 14:15:56.641293 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:56.641124 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31bff3ab-968d-4304-a5fb-3da783dc00f0-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-kw7tc\" (UID: \"31bff3ab-968d-4304-a5fb-3da783dc00f0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kw7tc" Nov 03 14:15:56.742491 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:56.742447 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p86v\" (UniqueName: \"kubernetes.io/projected/31bff3ab-968d-4304-a5fb-3da783dc00f0-kube-api-access-4p86v\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-kw7tc\" (UID: \"31bff3ab-968d-4304-a5fb-3da783dc00f0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kw7tc" Nov 03 14:15:56.742687 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:56.742504 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31bff3ab-968d-4304-a5fb-3da783dc00f0-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-kw7tc\" (UID: \"31bff3ab-968d-4304-a5fb-3da783dc00f0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kw7tc" Nov 03 14:15:56.742947 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:56.742929 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31bff3ab-968d-4304-a5fb-3da783dc00f0-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-kw7tc\" (UID: \"31bff3ab-968d-4304-a5fb-3da783dc00f0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kw7tc" Nov 03 14:15:56.764727 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:56.764696 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p86v\" (UniqueName: \"kubernetes.io/projected/31bff3ab-968d-4304-a5fb-3da783dc00f0-kube-api-access-4p86v\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-kw7tc\" (UID: \"31bff3ab-968d-4304-a5fb-3da783dc00f0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kw7tc" Nov 03 14:15:56.842986 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:56.842912 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kw7tc" Nov 03 14:15:57.192239 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:57.192170 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kw7tc"] Nov 03 14:15:57.195936 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:15:57.195905 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31bff3ab_968d_4304_a5fb_3da783dc00f0.slice/crio-03313b4da4588c8109723b91c20f794785e82d068a8f93cad7d2c4d47c71a1e0 WatchSource:0}: Error finding container 03313b4da4588c8109723b91c20f794785e82d068a8f93cad7d2c4d47c71a1e0: Status 404 returned error can't find the container with id 03313b4da4588c8109723b91c20f794785e82d068a8f93cad7d2c4d47c71a1e0 Nov 03 14:15:57.681910 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:57.681872 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kw7tc" event={"ID":"31bff3ab-968d-4304-a5fb-3da783dc00f0","Type":"ContainerStarted","Data":"03313b4da4588c8109723b91c20f794785e82d068a8f93cad7d2c4d47c71a1e0"} Nov 03 14:15:57.683357 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:57.683333 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-t7s86" event={"ID":"57e263ee-fe23-475e-aa0d-72f40f837809","Type":"ContainerStarted","Data":"c4f07b520da3bd7a04c74f65863eb7f4d61f1f757724ea2371145ade9a51f1f1"} Nov 03 14:15:57.683489 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:57.683473 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-t7s86" Nov 03 14:15:57.733678 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:57.733635 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-t7s86" podStartSLOduration=2.113826316 podStartE2EDuration="4.733622696s" totalImagesPullingTime="2.61979638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:15:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:15:57.733319664 +0000 UTC m=+291.622109256" watchObservedRunningTime="2026-11-03 14:15:57.733622696 +0000 UTC m=+291.622412268" Nov 03 14:15:59.694890 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:59.694830 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kw7tc" event={"ID":"31bff3ab-968d-4304-a5fb-3da783dc00f0","Type":"ContainerStarted","Data":"0c20aec4e279ac7a395c0bdd46153d6bced7dc011414082116aadbc18427b044"} Nov 03 14:15:59.770053 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:15:59.770002 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kw7tc" podStartSLOduration=2.004803868 podStartE2EDuration="3.769990501s" totalImagesPullingTime="1.765186633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:15:59.756568061 +0000 UTC m=+293.645357655" watchObservedRunningTime="2026-11-03 14:15:59.769990501 +0000 UTC m=+293.658780082" Nov 03 14:16:01.535611 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:01.535576 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5zmt7"] Nov 03 14:16:01.539322 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:01.539300 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5zmt7" Nov 03 14:16:01.574687 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:01.574658 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5zmt7"] Nov 03 14:16:01.584403 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:01.584372 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf1af3ea-b193-4ba8-ada7-a76feef7e408-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-5zmt7\" (UID: \"cf1af3ea-b193-4ba8-ada7-a76feef7e408\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5zmt7" Nov 03 14:16:01.584544 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:01.584407 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chvwk\" (UniqueName: \"kubernetes.io/projected/cf1af3ea-b193-4ba8-ada7-a76feef7e408-kube-api-access-chvwk\") pod \"kuadrant-operator-controller-manager-cf8469d7b-5zmt7\" (UID: \"cf1af3ea-b193-4ba8-ada7-a76feef7e408\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5zmt7" Nov 03 14:16:01.650765 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:01.650731 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-47h52\"" Nov 03 14:16:01.685488 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:01.685453 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf1af3ea-b193-4ba8-ada7-a76feef7e408-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-5zmt7\" (UID: \"cf1af3ea-b193-4ba8-ada7-a76feef7e408\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5zmt7" Nov 03 14:16:01.685488 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:01.685492 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chvwk\" (UniqueName: \"kubernetes.io/projected/cf1af3ea-b193-4ba8-ada7-a76feef7e408-kube-api-access-chvwk\") pod \"kuadrant-operator-controller-manager-cf8469d7b-5zmt7\" (UID: \"cf1af3ea-b193-4ba8-ada7-a76feef7e408\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5zmt7" Nov 03 14:16:01.685934 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:01.685910 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf1af3ea-b193-4ba8-ada7-a76feef7e408-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-5zmt7\" (UID: \"cf1af3ea-b193-4ba8-ada7-a76feef7e408\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5zmt7" Nov 03 14:16:01.724669 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:01.724636 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chvwk\" (UniqueName: \"kubernetes.io/projected/cf1af3ea-b193-4ba8-ada7-a76feef7e408-kube-api-access-chvwk\") pod \"kuadrant-operator-controller-manager-cf8469d7b-5zmt7\" (UID: \"cf1af3ea-b193-4ba8-ada7-a76feef7e408\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5zmt7" Nov 03 14:16:01.852361 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:01.852279 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5zmt7" Nov 03 14:16:02.188759 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:02.188727 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5zmt7"] Nov 03 14:16:02.191506 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:16:02.191468 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf1af3ea_b193_4ba8_ada7_a76feef7e408.slice/crio-ef71db8d229b40741bf339866e5dd78e4582ad450d0b6ffd1d95743eedf941b2 WatchSource:0}: Error finding container ef71db8d229b40741bf339866e5dd78e4582ad450d0b6ffd1d95743eedf941b2: Status 404 returned error can't find the container with id ef71db8d229b40741bf339866e5dd78e4582ad450d0b6ffd1d95743eedf941b2 Nov 03 14:16:02.710774 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:02.710737 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5zmt7" event={"ID":"cf1af3ea-b193-4ba8-ada7-a76feef7e408","Type":"ContainerStarted","Data":"ef71db8d229b40741bf339866e5dd78e4582ad450d0b6ffd1d95743eedf941b2"} Nov 03 14:16:05.946085 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:05.946048 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-jf2qh"] Nov 03 14:16:05.950210 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:05.950186 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-jf2qh" Nov 03 14:16:05.967318 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:05.967293 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-q8m55\"" Nov 03 14:16:05.981740 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:05.981710 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-jf2qh"] Nov 03 14:16:06.024565 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:06.024524 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g5zz\" (UniqueName: \"kubernetes.io/projected/37e492af-6e43-4224-be07-6e92f69bbae1-kube-api-access-7g5zz\") pod \"authorino-operator-7b7bb4bc77-jf2qh\" (UID: \"37e492af-6e43-4224-be07-6e92f69bbae1\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-jf2qh" Nov 03 14:16:06.125975 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:06.125937 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g5zz\" (UniqueName: \"kubernetes.io/projected/37e492af-6e43-4224-be07-6e92f69bbae1-kube-api-access-7g5zz\") pod \"authorino-operator-7b7bb4bc77-jf2qh\" (UID: \"37e492af-6e43-4224-be07-6e92f69bbae1\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-jf2qh" Nov 03 14:16:06.151789 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:06.151762 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g5zz\" (UniqueName: \"kubernetes.io/projected/37e492af-6e43-4224-be07-6e92f69bbae1-kube-api-access-7g5zz\") pod \"authorino-operator-7b7bb4bc77-jf2qh\" (UID: \"37e492af-6e43-4224-be07-6e92f69bbae1\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-jf2qh" Nov 03 14:16:06.270361 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:06.270333 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/3.log" Nov 03 14:16:06.270530 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:06.270403 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/3.log" Nov 03 14:16:06.279869 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:06.279829 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-q8m55\"" Nov 03 14:16:06.285446 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:06.285430 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-jf2qh" Nov 03 14:16:06.594277 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:06.594245 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-jf2qh"] Nov 03 14:16:06.597411 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:16:06.597381 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37e492af_6e43_4224_be07_6e92f69bbae1.slice/crio-c7131579f10597f3f0c9083b5426561bd9096a58a3e0a2132d3fdbace3512477 WatchSource:0}: Error finding container c7131579f10597f3f0c9083b5426561bd9096a58a3e0a2132d3fdbace3512477: Status 404 returned error can't find the container with id c7131579f10597f3f0c9083b5426561bd9096a58a3e0a2132d3fdbace3512477 Nov 03 14:16:06.731998 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:06.731959 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5zmt7" event={"ID":"cf1af3ea-b193-4ba8-ada7-a76feef7e408","Type":"ContainerStarted","Data":"15f088383d343ed1add345be934467dfb63e925c485da64c08309599af3b1913"} Nov 03 14:16:06.732168 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:06.732053 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5zmt7" Nov 03 14:16:06.733319 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:06.733293 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-jf2qh" event={"ID":"37e492af-6e43-4224-be07-6e92f69bbae1","Type":"ContainerStarted","Data":"c7131579f10597f3f0c9083b5426561bd9096a58a3e0a2132d3fdbace3512477"} Nov 03 14:16:06.794429 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:06.794381 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5zmt7" podStartSLOduration=1.849204493 podStartE2EDuration="5.794368281s" totalImagesPullingTime="3.945163788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:16:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:16:06.794169394 +0000 UTC m=+300.682958975" watchObservedRunningTime="2026-11-03 14:16:06.794368281 +0000 UTC m=+300.683157862" Nov 03 14:16:08.691185 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:08.691153 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-t7s86" Nov 03 14:16:09.439429 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:16:09.439392 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:16:09.544558 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:09.544526 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-vnxfr"] Nov 03 14:16:09.590396 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:09.590367 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-vnxfr" Nov 03 14:16:09.608729 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:09.608693 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-vnxfr"] Nov 03 14:16:09.655270 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:09.655235 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6d072299-2ec4-4321-ac7f-7a46844bd383-apiservice-cert\") pod \"opendatahub-operator-controller-manager-867995fc99-vnxfr\" (UID: \"6d072299-2ec4-4321-ac7f-7a46844bd383\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-vnxfr" Nov 03 14:16:09.655270 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:09.655272 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6d072299-2ec4-4321-ac7f-7a46844bd383-manifests\") pod \"opendatahub-operator-controller-manager-867995fc99-vnxfr\" (UID: \"6d072299-2ec4-4321-ac7f-7a46844bd383\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-vnxfr" Nov 03 14:16:09.655499 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:09.655297 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d072299-2ec4-4321-ac7f-7a46844bd383-metrics-cert\") pod \"opendatahub-operator-controller-manager-867995fc99-vnxfr\" (UID: \"6d072299-2ec4-4321-ac7f-7a46844bd383\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-vnxfr" Nov 03 14:16:09.655499 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:09.655408 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csc5g\" (UniqueName: \"kubernetes.io/projected/6d072299-2ec4-4321-ac7f-7a46844bd383-kube-api-access-csc5g\") pod \"opendatahub-operator-controller-manager-867995fc99-vnxfr\" (UID: \"6d072299-2ec4-4321-ac7f-7a46844bd383\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-vnxfr" Nov 03 14:16:09.655499 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:09.655430 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d072299-2ec4-4321-ac7f-7a46844bd383-tmp\") pod \"opendatahub-operator-controller-manager-867995fc99-vnxfr\" (UID: \"6d072299-2ec4-4321-ac7f-7a46844bd383\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-vnxfr" Nov 03 14:16:09.655499 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:09.655465 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6d072299-2ec4-4321-ac7f-7a46844bd383-webhook-cert\") pod \"opendatahub-operator-controller-manager-867995fc99-vnxfr\" (UID: \"6d072299-2ec4-4321-ac7f-7a46844bd383\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-vnxfr" Nov 03 14:16:09.672024 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:09.671994 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 14:16:09.672024 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:09.672013 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 14:16:09.712602 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:09.712526 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 14:16:09.713063 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:09.712963 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 14:16:09.713809 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:09.713786 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 14:16:09.713911 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:09.713851 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-m8l2v\"" Nov 03 14:16:09.747970 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:09.747939 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-jf2qh" event={"ID":"37e492af-6e43-4224-be07-6e92f69bbae1","Type":"ContainerStarted","Data":"a1558ada292921088eff41ea45455f48e3effe4637207819d2fa4961335730f1"} Nov 03 14:16:09.748088 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:09.748026 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-jf2qh" Nov 03 14:16:09.756466 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:09.756445 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6d072299-2ec4-4321-ac7f-7a46844bd383-webhook-cert\") pod \"opendatahub-operator-controller-manager-867995fc99-vnxfr\" (UID: \"6d072299-2ec4-4321-ac7f-7a46844bd383\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-vnxfr" Nov 03 14:16:09.756571 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:09.756487 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6d072299-2ec4-4321-ac7f-7a46844bd383-apiservice-cert\") pod \"opendatahub-operator-controller-manager-867995fc99-vnxfr\" (UID: \"6d072299-2ec4-4321-ac7f-7a46844bd383\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-vnxfr" Nov 03 14:16:09.756571 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:09.756508 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6d072299-2ec4-4321-ac7f-7a46844bd383-manifests\") pod \"opendatahub-operator-controller-manager-867995fc99-vnxfr\" (UID: \"6d072299-2ec4-4321-ac7f-7a46844bd383\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-vnxfr" Nov 03 14:16:09.756571 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:09.756538 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d072299-2ec4-4321-ac7f-7a46844bd383-metrics-cert\") pod \"opendatahub-operator-controller-manager-867995fc99-vnxfr\" (UID: \"6d072299-2ec4-4321-ac7f-7a46844bd383\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-vnxfr" Nov 03 14:16:09.756724 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:09.756589 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csc5g\" (UniqueName: \"kubernetes.io/projected/6d072299-2ec4-4321-ac7f-7a46844bd383-kube-api-access-csc5g\") pod \"opendatahub-operator-controller-manager-867995fc99-vnxfr\" (UID: \"6d072299-2ec4-4321-ac7f-7a46844bd383\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-vnxfr" Nov 03 14:16:09.756724 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:09.756627 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d072299-2ec4-4321-ac7f-7a46844bd383-tmp\") pod \"opendatahub-operator-controller-manager-867995fc99-vnxfr\" (UID: \"6d072299-2ec4-4321-ac7f-7a46844bd383\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-vnxfr" Nov 03 14:16:09.757026 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:09.756995 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6d072299-2ec4-4321-ac7f-7a46844bd383-manifests\") pod \"opendatahub-operator-controller-manager-867995fc99-vnxfr\" (UID: \"6d072299-2ec4-4321-ac7f-7a46844bd383\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-vnxfr" Nov 03 14:16:09.759046 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:09.759024 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d072299-2ec4-4321-ac7f-7a46844bd383-tmp\") pod \"opendatahub-operator-controller-manager-867995fc99-vnxfr\" (UID: \"6d072299-2ec4-4321-ac7f-7a46844bd383\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-vnxfr" Nov 03 14:16:09.759196 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:09.759178 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6d072299-2ec4-4321-ac7f-7a46844bd383-webhook-cert\") pod \"opendatahub-operator-controller-manager-867995fc99-vnxfr\" (UID: \"6d072299-2ec4-4321-ac7f-7a46844bd383\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-vnxfr" Nov 03 14:16:09.759374 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:09.759354 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d072299-2ec4-4321-ac7f-7a46844bd383-metrics-cert\") pod \"opendatahub-operator-controller-manager-867995fc99-vnxfr\" (UID: \"6d072299-2ec4-4321-ac7f-7a46844bd383\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-vnxfr" Nov 03 14:16:09.759434 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:09.759358 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6d072299-2ec4-4321-ac7f-7a46844bd383-apiservice-cert\") pod \"opendatahub-operator-controller-manager-867995fc99-vnxfr\" (UID: \"6d072299-2ec4-4321-ac7f-7a46844bd383\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-vnxfr" Nov 03 14:16:09.783379 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:09.783355 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csc5g\" (UniqueName: \"kubernetes.io/projected/6d072299-2ec4-4321-ac7f-7a46844bd383-kube-api-access-csc5g\") pod \"opendatahub-operator-controller-manager-867995fc99-vnxfr\" (UID: \"6d072299-2ec4-4321-ac7f-7a46844bd383\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-vnxfr" Nov 03 14:16:09.805418 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:09.805372 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-jf2qh" podStartSLOduration=2.547831576 podStartE2EDuration="4.805358508s" totalImagesPullingTime="2.257526932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:16:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:16:09.804903605 +0000 UTC m=+303.693693183" watchObservedRunningTime="2026-11-03 14:16:09.805358508 +0000 UTC m=+303.694148088" Nov 03 14:16:09.904723 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:09.904688 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-vnxfr" Nov 03 14:16:10.259978 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:10.259943 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-vnxfr"] Nov 03 14:16:10.263232 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:16:10.263203 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d072299_2ec4_4321_ac7f_7a46844bd383.slice/crio-579e2a42a4c28a95ef806c5a2b3d4e0a7a71cec0a0b0a00cf565b84847aaae5f WatchSource:0}: Error finding container 579e2a42a4c28a95ef806c5a2b3d4e0a7a71cec0a0b0a00cf565b84847aaae5f: Status 404 returned error can't find the container with id 579e2a42a4c28a95ef806c5a2b3d4e0a7a71cec0a0b0a00cf565b84847aaae5f Nov 03 14:16:10.268187 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:10.268156 5060 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 14:16:10.545120 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:10.544997 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-569f5"] Nov 03 14:16:10.548891 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:10.548867 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-569f5" Nov 03 14:16:10.571203 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:10.571178 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-xxbmm\"" Nov 03 14:16:10.572727 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:10.572700 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-569f5"] Nov 03 14:16:10.663603 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:10.663570 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq246\" (UniqueName: \"kubernetes.io/projected/3ced846c-60f0-4206-836e-f94c281d6438-kube-api-access-mq246\") pod \"limitador-operator-controller-manager-5778bb4449-569f5\" (UID: \"3ced846c-60f0-4206-836e-f94c281d6438\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-569f5" Nov 03 14:16:10.753335 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:10.753297 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-vnxfr" event={"ID":"6d072299-2ec4-4321-ac7f-7a46844bd383","Type":"ContainerStarted","Data":"579e2a42a4c28a95ef806c5a2b3d4e0a7a71cec0a0b0a00cf565b84847aaae5f"} Nov 03 14:16:10.764965 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:10.764938 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq246\" (UniqueName: \"kubernetes.io/projected/3ced846c-60f0-4206-836e-f94c281d6438-kube-api-access-mq246\") pod \"limitador-operator-controller-manager-5778bb4449-569f5\" (UID: \"3ced846c-60f0-4206-836e-f94c281d6438\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-569f5" Nov 03 14:16:10.789146 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:10.789086 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq246\" (UniqueName: \"kubernetes.io/projected/3ced846c-60f0-4206-836e-f94c281d6438-kube-api-access-mq246\") pod \"limitador-operator-controller-manager-5778bb4449-569f5\" (UID: \"3ced846c-60f0-4206-836e-f94c281d6438\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-569f5" Nov 03 14:16:10.861935 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:10.861854 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-569f5" Nov 03 14:16:11.171184 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:11.171082 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-569f5"] Nov 03 14:16:11.174136 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:16:11.174107 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ced846c_60f0_4206_836e_f94c281d6438.slice/crio-c642be0ae8dafc9c5c0ea5fd9ea7880128cc155f5b322d13621272f6a2467b14 WatchSource:0}: Error finding container c642be0ae8dafc9c5c0ea5fd9ea7880128cc155f5b322d13621272f6a2467b14: Status 404 returned error can't find the container with id c642be0ae8dafc9c5c0ea5fd9ea7880128cc155f5b322d13621272f6a2467b14 Nov 03 14:16:11.760610 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:11.760569 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-569f5" event={"ID":"3ced846c-60f0-4206-836e-f94c281d6438","Type":"ContainerStarted","Data":"c642be0ae8dafc9c5c0ea5fd9ea7880128cc155f5b322d13621272f6a2467b14"} Nov 03 14:16:13.776784 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:13.776747 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-569f5" event={"ID":"3ced846c-60f0-4206-836e-f94c281d6438","Type":"ContainerStarted","Data":"de1d0f342a7847b5b28c98bfe7c7105da904f58a781ed25b40f73f3d17c90dd4"} Nov 03 14:16:13.777242 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:13.776939 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-569f5" Nov 03 14:16:13.778510 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:13.778478 5060 generic.go:356] "Generic (PLEG): container finished" podID="6d072299-2ec4-4321-ac7f-7a46844bd383" containerID="5b32b2abb45ee3f0d17e4a326f8a7496581cccb4c6148b582583d170848614e8" exitCode=0 Nov 03 14:16:13.778624 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:13.778563 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-vnxfr" event={"ID":"6d072299-2ec4-4321-ac7f-7a46844bd383","Type":"ContainerDied","Data":"5b32b2abb45ee3f0d17e4a326f8a7496581cccb4c6148b582583d170848614e8"} Nov 03 14:16:13.821800 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:13.821701 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-569f5" podStartSLOduration=2.256402021 podStartE2EDuration="3.821686218s" totalImagesPullingTime="1.565284197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:16:13.820165602 +0000 UTC m=+307.708955182" watchObservedRunningTime="2026-11-03 14:16:13.821686218 +0000 UTC m=+307.710475801" Nov 03 14:16:14.785623 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:14.785585 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-vnxfr" event={"ID":"6d072299-2ec4-4321-ac7f-7a46844bd383","Type":"ContainerStarted","Data":"cf985978fe81da9c6385febf1d7815eb70612fbf50c4277db2e5999c8e21d9f6"} Nov 03 14:16:14.871935 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:14.871820 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-vnxfr" podStartSLOduration=2.188704844 podStartE2EDuration="5.871802604s" totalImagesPullingTime="3.208469043s" totalInitContainerRuntime="474.628717ms" isStatefulPod=false podCreationTimestamp="2026-11-03 14:16:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:16:14.858477497 +0000 UTC m=+308.747267077" watchObservedRunningTime="2026-11-03 14:16:14.871802604 +0000 UTC m=+308.760592184" Nov 03 14:16:15.790608 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:15.790577 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-vnxfr" Nov 03 14:16:17.740349 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:17.740264 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5zmt7" Nov 03 14:16:18.327588 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:18.327552 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-82pkj"] Nov 03 14:16:18.331179 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:18.331157 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-82pkj" Nov 03 14:16:18.362757 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:18.362730 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-82pkj"] Nov 03 14:16:18.432963 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:18.432929 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzgpl\" (UniqueName: \"kubernetes.io/projected/b0d2f12e-9c6c-4e22-ba6f-197f411b0666-kube-api-access-jzgpl\") pod \"kuadrant-operator-controller-manager-678d87766f-82pkj\" (UID: \"b0d2f12e-9c6c-4e22-ba6f-197f411b0666\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-82pkj" Nov 03 14:16:18.433144 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:18.432983 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0d2f12e-9c6c-4e22-ba6f-197f411b0666-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-82pkj\" (UID: \"b0d2f12e-9c6c-4e22-ba6f-197f411b0666\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-82pkj" Nov 03 14:16:18.533445 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:18.533411 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0d2f12e-9c6c-4e22-ba6f-197f411b0666-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-82pkj\" (UID: \"b0d2f12e-9c6c-4e22-ba6f-197f411b0666\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-82pkj" Nov 03 14:16:18.533635 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:18.533507 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzgpl\" (UniqueName: \"kubernetes.io/projected/b0d2f12e-9c6c-4e22-ba6f-197f411b0666-kube-api-access-jzgpl\") pod \"kuadrant-operator-controller-manager-678d87766f-82pkj\" (UID: \"b0d2f12e-9c6c-4e22-ba6f-197f411b0666\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-82pkj" Nov 03 14:16:18.533830 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:18.533806 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0d2f12e-9c6c-4e22-ba6f-197f411b0666-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-82pkj\" (UID: \"b0d2f12e-9c6c-4e22-ba6f-197f411b0666\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-82pkj" Nov 03 14:16:18.556348 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:18.556315 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzgpl\" (UniqueName: \"kubernetes.io/projected/b0d2f12e-9c6c-4e22-ba6f-197f411b0666-kube-api-access-jzgpl\") pod \"kuadrant-operator-controller-manager-678d87766f-82pkj\" (UID: \"b0d2f12e-9c6c-4e22-ba6f-197f411b0666\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-82pkj" Nov 03 14:16:18.644427 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:18.644345 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-82pkj" Nov 03 14:16:18.971681 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:18.971654 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-82pkj"] Nov 03 14:16:18.972796 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:16:18.972771 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0d2f12e_9c6c_4e22_ba6f_197f411b0666.slice/crio-74cfecfd02be6266f26c1009dbf65b1459b0b11cb877cde286aa3ab5a13aef97 WatchSource:0}: Error finding container 74cfecfd02be6266f26c1009dbf65b1459b0b11cb877cde286aa3ab5a13aef97: Status 404 returned error can't find the container with id 74cfecfd02be6266f26c1009dbf65b1459b0b11cb877cde286aa3ab5a13aef97 Nov 03 14:16:19.810817 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:19.810781 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-82pkj" event={"ID":"b0d2f12e-9c6c-4e22-ba6f-197f411b0666","Type":"ContainerStarted","Data":"ef04332a24640b1550a8bd7c96589061aa07f08c6ee0ca1942f3d5c5e90f1039"} Nov 03 14:16:19.811003 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:19.810851 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-82pkj" event={"ID":"b0d2f12e-9c6c-4e22-ba6f-197f411b0666","Type":"ContainerStarted","Data":"74cfecfd02be6266f26c1009dbf65b1459b0b11cb877cde286aa3ab5a13aef97"} Nov 03 14:16:19.811003 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:19.810876 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-82pkj" Nov 03 14:16:19.885274 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:19.885217 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-82pkj" podStartSLOduration=1.8852006000000001 podStartE2EDuration="1.8852006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:16:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:16:19.871523635 +0000 UTC m=+313.760313215" watchObservedRunningTime="2026-11-03 14:16:19.8852006 +0000 UTC m=+313.773990181" Nov 03 14:16:20.755812 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:20.755783 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-jf2qh" Nov 03 14:16:24.788742 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:24.788711 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-569f5" Nov 03 14:16:26.796984 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:26.796949 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-vnxfr" Nov 03 14:16:30.819190 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:30.819152 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-82pkj" Nov 03 14:16:30.932440 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:30.932407 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5zmt7"] Nov 03 14:16:30.932634 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:30.932602 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5zmt7" podUID="cf1af3ea-b193-4ba8-ada7-a76feef7e408" containerName="manager" containerID="cri-o://15f088383d343ed1add345be934467dfb63e925c485da64c08309599af3b1913" gracePeriod=10 Nov 03 14:16:31.214966 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:31.214935 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5zmt7" Nov 03 14:16:31.342653 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:31.342556 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf1af3ea-b193-4ba8-ada7-a76feef7e408-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf1af3ea-b193-4ba8-ada7-a76feef7e408-extensions-socket-volume\") pod \"cf1af3ea-b193-4ba8-ada7-a76feef7e408\" (UID: \"cf1af3ea-b193-4ba8-ada7-a76feef7e408\") " Nov 03 14:16:31.342804 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:31.342660 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf1af3ea-b193-4ba8-ada7-a76feef7e408-kube-api-access-chvwk\" (UniqueName: \"kubernetes.io/projected/cf1af3ea-b193-4ba8-ada7-a76feef7e408-kube-api-access-chvwk\") pod \"cf1af3ea-b193-4ba8-ada7-a76feef7e408\" (UID: \"cf1af3ea-b193-4ba8-ada7-a76feef7e408\") " Nov 03 14:16:31.343102 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:31.343077 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf1af3ea-b193-4ba8-ada7-a76feef7e408-extensions-socket-volume" pod "cf1af3ea-b193-4ba8-ada7-a76feef7e408" (UID: "cf1af3ea-b193-4ba8-ada7-a76feef7e408"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:16:31.344996 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:31.344971 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf1af3ea-b193-4ba8-ada7-a76feef7e408-kube-api-access-chvwk" pod "cf1af3ea-b193-4ba8-ada7-a76feef7e408" (UID: "cf1af3ea-b193-4ba8-ada7-a76feef7e408"). InnerVolumeSpecName "kube-api-access-chvwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:16:31.443323 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:31.443292 5060 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf1af3ea-b193-4ba8-ada7-a76feef7e408-extensions-socket-volume\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:16:31.443323 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:31.443319 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chvwk\" (UniqueName: \"kubernetes.io/projected/cf1af3ea-b193-4ba8-ada7-a76feef7e408-kube-api-access-chvwk\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:16:31.869968 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:31.869935 5060 generic.go:356] "Generic (PLEG): container finished" podID="cf1af3ea-b193-4ba8-ada7-a76feef7e408" containerID="15f088383d343ed1add345be934467dfb63e925c485da64c08309599af3b1913" exitCode=0 Nov 03 14:16:31.870373 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:31.869984 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5zmt7" Nov 03 14:16:31.870373 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:31.870028 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5zmt7" event={"ID":"cf1af3ea-b193-4ba8-ada7-a76feef7e408","Type":"ContainerDied","Data":"15f088383d343ed1add345be934467dfb63e925c485da64c08309599af3b1913"} Nov 03 14:16:31.870373 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:31.870083 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5zmt7" event={"ID":"cf1af3ea-b193-4ba8-ada7-a76feef7e408","Type":"ContainerDied","Data":"ef71db8d229b40741bf339866e5dd78e4582ad450d0b6ffd1d95743eedf941b2"} Nov 03 14:16:31.870373 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:31.870106 5060 scope.go:122] "RemoveContainer" containerID="15f088383d343ed1add345be934467dfb63e925c485da64c08309599af3b1913" Nov 03 14:16:31.879801 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:31.879785 5060 scope.go:122] "RemoveContainer" containerID="15f088383d343ed1add345be934467dfb63e925c485da64c08309599af3b1913" Nov 03 14:16:31.880210 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:16:31.880189 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15f088383d343ed1add345be934467dfb63e925c485da64c08309599af3b1913\": container with ID starting with 15f088383d343ed1add345be934467dfb63e925c485da64c08309599af3b1913 not found: ID does not exist" containerID="15f088383d343ed1add345be934467dfb63e925c485da64c08309599af3b1913" Nov 03 14:16:31.880262 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:31.880229 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15f088383d343ed1add345be934467dfb63e925c485da64c08309599af3b1913"} err="failed to get container status \"15f088383d343ed1add345be934467dfb63e925c485da64c08309599af3b1913\": rpc error: code = NotFound desc = could not find container \"15f088383d343ed1add345be934467dfb63e925c485da64c08309599af3b1913\": container with ID starting with 15f088383d343ed1add345be934467dfb63e925c485da64c08309599af3b1913 not found: ID does not exist" Nov 03 14:16:31.941376 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:31.941343 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5zmt7"] Nov 03 14:16:31.954449 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:31.954420 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5zmt7"] Nov 03 14:16:32.247625 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:32.247595 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf1af3ea-b193-4ba8-ada7-a76feef7e408" path="/var/lib/kubelet/pods/cf1af3ea-b193-4ba8-ada7-a76feef7e408/volumes" Nov 03 14:16:39.442065 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:16:39.442027 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:16:41.665749 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:41.665714 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-58f856b7f7-px6s2"] Nov 03 14:16:41.666151 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:41.666082 5060 cpu_manager.go:414] "containerMap: removing container" podUID="cf1af3ea-b193-4ba8-ada7-a76feef7e408" containerName="manager" Nov 03 14:16:41.666151 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:41.666093 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf1af3ea-b193-4ba8-ada7-a76feef7e408" containerName="manager" Nov 03 14:16:41.666225 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:41.666167 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf1af3ea-b193-4ba8-ada7-a76feef7e408" containerName="manager" Nov 03 14:16:41.669562 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:41.669540 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-58f856b7f7-px6s2" Nov 03 14:16:41.700246 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:41.700209 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 03 14:16:41.700246 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:41.700211 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-t754j\"" Nov 03 14:16:41.700594 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:41.700565 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 03 14:16:41.700716 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:41.700616 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 03 14:16:41.700716 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:41.700667 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 03 14:16:41.700716 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:41.700690 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 03 14:16:41.705010 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:41.704988 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-58f856b7f7-px6s2"] Nov 03 14:16:41.726714 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:41.726684 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvknn\" (UniqueName: \"kubernetes.io/projected/dacba53e-72c5-402e-88b8-aa0a4b088a9c-kube-api-access-zvknn\") pod \"jobset-controller-manager-58f856b7f7-px6s2\" (UID: \"dacba53e-72c5-402e-88b8-aa0a4b088a9c\") " pod="openshift-jobset-operator/jobset-controller-manager-58f856b7f7-px6s2" Nov 03 14:16:41.726948 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:41.726733 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dacba53e-72c5-402e-88b8-aa0a4b088a9c-cert\") pod \"jobset-controller-manager-58f856b7f7-px6s2\" (UID: \"dacba53e-72c5-402e-88b8-aa0a4b088a9c\") " pod="openshift-jobset-operator/jobset-controller-manager-58f856b7f7-px6s2" Nov 03 14:16:41.726948 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:41.726893 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/dacba53e-72c5-402e-88b8-aa0a4b088a9c-manager-config\") pod \"jobset-controller-manager-58f856b7f7-px6s2\" (UID: \"dacba53e-72c5-402e-88b8-aa0a4b088a9c\") " pod="openshift-jobset-operator/jobset-controller-manager-58f856b7f7-px6s2" Nov 03 14:16:41.727066 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:41.726963 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dacba53e-72c5-402e-88b8-aa0a4b088a9c-metrics-certs\") pod \"jobset-controller-manager-58f856b7f7-px6s2\" (UID: \"dacba53e-72c5-402e-88b8-aa0a4b088a9c\") " pod="openshift-jobset-operator/jobset-controller-manager-58f856b7f7-px6s2" Nov 03 14:16:41.827555 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:41.827520 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvknn\" (UniqueName: \"kubernetes.io/projected/dacba53e-72c5-402e-88b8-aa0a4b088a9c-kube-api-access-zvknn\") pod \"jobset-controller-manager-58f856b7f7-px6s2\" (UID: \"dacba53e-72c5-402e-88b8-aa0a4b088a9c\") " pod="openshift-jobset-operator/jobset-controller-manager-58f856b7f7-px6s2" Nov 03 14:16:41.827725 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:41.827592 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dacba53e-72c5-402e-88b8-aa0a4b088a9c-cert\") pod \"jobset-controller-manager-58f856b7f7-px6s2\" (UID: \"dacba53e-72c5-402e-88b8-aa0a4b088a9c\") " pod="openshift-jobset-operator/jobset-controller-manager-58f856b7f7-px6s2" Nov 03 14:16:41.827725 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:41.827656 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/dacba53e-72c5-402e-88b8-aa0a4b088a9c-manager-config\") pod \"jobset-controller-manager-58f856b7f7-px6s2\" (UID: \"dacba53e-72c5-402e-88b8-aa0a4b088a9c\") " pod="openshift-jobset-operator/jobset-controller-manager-58f856b7f7-px6s2" Nov 03 14:16:41.827857 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:41.827742 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dacba53e-72c5-402e-88b8-aa0a4b088a9c-metrics-certs\") pod \"jobset-controller-manager-58f856b7f7-px6s2\" (UID: \"dacba53e-72c5-402e-88b8-aa0a4b088a9c\") " pod="openshift-jobset-operator/jobset-controller-manager-58f856b7f7-px6s2" Nov 03 14:16:41.828317 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:41.828281 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/dacba53e-72c5-402e-88b8-aa0a4b088a9c-manager-config\") pod \"jobset-controller-manager-58f856b7f7-px6s2\" (UID: \"dacba53e-72c5-402e-88b8-aa0a4b088a9c\") " pod="openshift-jobset-operator/jobset-controller-manager-58f856b7f7-px6s2" Nov 03 14:16:41.830249 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:41.830225 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dacba53e-72c5-402e-88b8-aa0a4b088a9c-cert\") pod \"jobset-controller-manager-58f856b7f7-px6s2\" (UID: \"dacba53e-72c5-402e-88b8-aa0a4b088a9c\") " pod="openshift-jobset-operator/jobset-controller-manager-58f856b7f7-px6s2" Nov 03 14:16:41.830346 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:41.830276 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dacba53e-72c5-402e-88b8-aa0a4b088a9c-metrics-certs\") pod \"jobset-controller-manager-58f856b7f7-px6s2\" (UID: \"dacba53e-72c5-402e-88b8-aa0a4b088a9c\") " pod="openshift-jobset-operator/jobset-controller-manager-58f856b7f7-px6s2" Nov 03 14:16:41.854649 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:41.854627 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvknn\" (UniqueName: \"kubernetes.io/projected/dacba53e-72c5-402e-88b8-aa0a4b088a9c-kube-api-access-zvknn\") pod \"jobset-controller-manager-58f856b7f7-px6s2\" (UID: \"dacba53e-72c5-402e-88b8-aa0a4b088a9c\") " pod="openshift-jobset-operator/jobset-controller-manager-58f856b7f7-px6s2" Nov 03 14:16:41.981685 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:41.981600 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-58f856b7f7-px6s2" Nov 03 14:16:42.283605 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:42.283575 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-58f856b7f7-px6s2"] Nov 03 14:16:42.285971 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:16:42.285947 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddacba53e_72c5_402e_88b8_aa0a4b088a9c.slice/crio-8c5940b7f721af7487c817bbd0fc567536f96f6a301cf8383ff5e09d6ee6426e WatchSource:0}: Error finding container 8c5940b7f721af7487c817bbd0fc567536f96f6a301cf8383ff5e09d6ee6426e: Status 404 returned error can't find the container with id 8c5940b7f721af7487c817bbd0fc567536f96f6a301cf8383ff5e09d6ee6426e Nov 03 14:16:42.925121 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:42.925088 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-58f856b7f7-px6s2" event={"ID":"dacba53e-72c5-402e-88b8-aa0a4b088a9c","Type":"ContainerStarted","Data":"8c5940b7f721af7487c817bbd0fc567536f96f6a301cf8383ff5e09d6ee6426e"} Nov 03 14:16:45.944406 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:45.944300 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-58f856b7f7-px6s2" event={"ID":"dacba53e-72c5-402e-88b8-aa0a4b088a9c","Type":"ContainerStarted","Data":"4454a73d913295159c6b2e4b4be69319ada5fbc09cdd471abc941236f9ec4e9f"} Nov 03 14:16:45.944406 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:45.944405 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-58f856b7f7-px6s2" Nov 03 14:16:46.008320 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:46.008258 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-58f856b7f7-px6s2" podStartSLOduration=1.733055331 podStartE2EDuration="5.008245901s" totalImagesPullingTime="3.27519057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:16:46.00678743 +0000 UTC m=+339.895577012" watchObservedRunningTime="2026-11-03 14:16:46.008245901 +0000 UTC m=+339.897035481" Nov 03 14:16:56.954043 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:16:56.954009 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-58f856b7f7-px6s2" Nov 03 14:17:09.438752 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:17:09.438711 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:17:39.129789 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:17:39.129744 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-4ggmd"] Nov 03 14:17:39.133830 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:17:39.133809 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-4ggmd" Nov 03 14:17:39.164170 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:17:39.164143 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 14:17:39.164298 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:17:39.164225 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-qf9vn\"" Nov 03 14:17:39.164355 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:17:39.164330 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 03 14:17:39.165014 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:17:39.164988 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 14:17:39.166913 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:17:39.166890 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-4ggmd"] Nov 03 14:17:39.168459 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:17:39.168442 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-75cwz"] Nov 03 14:17:39.173909 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:17:39.172964 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-75cwz" Nov 03 14:17:39.185726 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:17:39.185702 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-75cwz"] Nov 03 14:17:39.200347 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:17:39.200320 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 03 14:17:39.200474 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:17:39.200441 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-njpm8\"" Nov 03 14:17:39.205037 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:17:39.205012 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6nj9\" (UniqueName: \"kubernetes.io/projected/7b8c342f-20c2-406f-a6a1-f8eef4118a6d-kube-api-access-l6nj9\") pod \"kserve-controller-manager-776cc66d47-4ggmd\" (UID: \"7b8c342f-20c2-406f-a6a1-f8eef4118a6d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-4ggmd" Nov 03 14:17:39.205151 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:17:39.205043 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7b8c342f-20c2-406f-a6a1-f8eef4118a6d-cert\") pod \"kserve-controller-manager-776cc66d47-4ggmd\" (UID: \"7b8c342f-20c2-406f-a6a1-f8eef4118a6d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-4ggmd" Nov 03 14:17:39.205151 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:17:39.205067 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzdjq\" (UniqueName: \"kubernetes.io/projected/d887bc8f-e13f-4853-b079-41e503286e1a-kube-api-access-gzdjq\") pod \"llmisvc-controller-manager-869d5dcc5d-75cwz\" (UID: \"d887bc8f-e13f-4853-b079-41e503286e1a\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-75cwz" Nov 03 14:17:39.205239 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:17:39.205159 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d887bc8f-e13f-4853-b079-41e503286e1a-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-75cwz\" (UID: \"d887bc8f-e13f-4853-b079-41e503286e1a\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-75cwz" Nov 03 14:17:39.305618 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:17:39.305586 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d887bc8f-e13f-4853-b079-41e503286e1a-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-75cwz\" (UID: \"d887bc8f-e13f-4853-b079-41e503286e1a\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-75cwz" Nov 03 14:17:39.305805 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:17:39.305676 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6nj9\" (UniqueName: \"kubernetes.io/projected/7b8c342f-20c2-406f-a6a1-f8eef4118a6d-kube-api-access-l6nj9\") pod \"kserve-controller-manager-776cc66d47-4ggmd\" (UID: \"7b8c342f-20c2-406f-a6a1-f8eef4118a6d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-4ggmd" Nov 03 14:17:39.305805 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:17:39.305703 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7b8c342f-20c2-406f-a6a1-f8eef4118a6d-cert\") pod \"kserve-controller-manager-776cc66d47-4ggmd\" (UID: \"7b8c342f-20c2-406f-a6a1-f8eef4118a6d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-4ggmd" Nov 03 14:17:39.305805 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:17:39.305737 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzdjq\" (UniqueName: \"kubernetes.io/projected/d887bc8f-e13f-4853-b079-41e503286e1a-kube-api-access-gzdjq\") pod \"llmisvc-controller-manager-869d5dcc5d-75cwz\" (UID: \"d887bc8f-e13f-4853-b079-41e503286e1a\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-75cwz" Nov 03 14:17:39.308118 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:17:39.308089 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d887bc8f-e13f-4853-b079-41e503286e1a-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-75cwz\" (UID: \"d887bc8f-e13f-4853-b079-41e503286e1a\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-75cwz" Nov 03 14:17:39.308281 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:17:39.308265 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7b8c342f-20c2-406f-a6a1-f8eef4118a6d-cert\") pod \"kserve-controller-manager-776cc66d47-4ggmd\" (UID: \"7b8c342f-20c2-406f-a6a1-f8eef4118a6d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-4ggmd" Nov 03 14:17:39.327285 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:17:39.327250 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6nj9\" (UniqueName: \"kubernetes.io/projected/7b8c342f-20c2-406f-a6a1-f8eef4118a6d-kube-api-access-l6nj9\") pod \"kserve-controller-manager-776cc66d47-4ggmd\" (UID: \"7b8c342f-20c2-406f-a6a1-f8eef4118a6d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-4ggmd" Nov 03 14:17:39.327377 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:17:39.327353 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzdjq\" (UniqueName: \"kubernetes.io/projected/d887bc8f-e13f-4853-b079-41e503286e1a-kube-api-access-gzdjq\") pod \"llmisvc-controller-manager-869d5dcc5d-75cwz\" (UID: \"d887bc8f-e13f-4853-b079-41e503286e1a\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-75cwz" Nov 03 14:17:39.437505 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:17:39.437419 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:17:39.446731 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:17:39.446696 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-4ggmd" Nov 03 14:17:39.495126 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:17:39.494897 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-75cwz" Nov 03 14:17:39.745270 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:17:39.745226 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-4ggmd"] Nov 03 14:17:39.747626 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:17:39.747600 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b8c342f_20c2_406f_a6a1_f8eef4118a6d.slice/crio-429bd1218368db33b87a5b3028b7c4673405d56cd1720975bd4e85f036d34146 WatchSource:0}: Error finding container 429bd1218368db33b87a5b3028b7c4673405d56cd1720975bd4e85f036d34146: Status 404 returned error can't find the container with id 429bd1218368db33b87a5b3028b7c4673405d56cd1720975bd4e85f036d34146 Nov 03 14:17:39.806375 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:17:39.806307 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-75cwz"] Nov 03 14:17:39.809160 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:17:39.809129 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd887bc8f_e13f_4853_b079_41e503286e1a.slice/crio-a1152f64802d266054f9e20fb194d5fe21e0d72c8d9e0cc6659d1dd5fa646ca4 WatchSource:0}: Error finding container a1152f64802d266054f9e20fb194d5fe21e0d72c8d9e0cc6659d1dd5fa646ca4: Status 404 returned error can't find the container with id a1152f64802d266054f9e20fb194d5fe21e0d72c8d9e0cc6659d1dd5fa646ca4 Nov 03 14:17:40.202942 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:17:40.202855 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-4ggmd" event={"ID":"7b8c342f-20c2-406f-a6a1-f8eef4118a6d","Type":"ContainerStarted","Data":"429bd1218368db33b87a5b3028b7c4673405d56cd1720975bd4e85f036d34146"} Nov 03 14:17:40.203927 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:17:40.203906 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-75cwz" event={"ID":"d887bc8f-e13f-4853-b079-41e503286e1a","Type":"ContainerStarted","Data":"a1152f64802d266054f9e20fb194d5fe21e0d72c8d9e0cc6659d1dd5fa646ca4"} Nov 03 14:17:43.225820 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:17:43.225727 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-75cwz" event={"ID":"d887bc8f-e13f-4853-b079-41e503286e1a","Type":"ContainerStarted","Data":"22c8f3a146e990afcb1aff942a1b3ca4f9a004c3170a384f3ad1189ee58ed5a0"} Nov 03 14:17:43.226272 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:17:43.225828 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-75cwz" Nov 03 14:17:43.227760 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:17:43.227733 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-4ggmd" event={"ID":"7b8c342f-20c2-406f-a6a1-f8eef4118a6d","Type":"ContainerStarted","Data":"c660d330c14d4c819755374caf3a8d3e27e7147a87148d011641b2eecbe5aba0"} Nov 03 14:17:43.227902 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:17:43.227871 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-4ggmd" Nov 03 14:17:43.271421 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:17:43.271366 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-75cwz" podStartSLOduration=1.2455049599999999 podStartE2EDuration="4.27135102s" totalImagesPullingTime="3.02584606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:17:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:17:43.269642462 +0000 UTC m=+397.158432046" watchObservedRunningTime="2026-11-03 14:17:43.27135102 +0000 UTC m=+397.160140601" Nov 03 14:17:43.315052 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:17:43.314988 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-4ggmd" podStartSLOduration=1.165594364 podStartE2EDuration="4.31496815s" totalImagesPullingTime="3.149373786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:17:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:17:43.313138225 +0000 UTC m=+397.201927808" watchObservedRunningTime="2026-11-03 14:17:43.31496815 +0000 UTC m=+397.203757735" Nov 03 14:18:09.438137 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:18:09.438091 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:18:14.235589 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:18:14.235558 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-75cwz" Nov 03 14:18:14.238398 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:18:14.238378 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-4ggmd" Nov 03 14:18:39.438588 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:18:39.438555 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:18:43.988172 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:18:43.988136 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-54d5w"] Nov 03 14:18:43.991889 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:18:43.991872 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-54d5w" Nov 03 14:18:44.015924 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:18:44.015895 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-54d5w"] Nov 03 14:18:44.085830 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:18:44.085788 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjs26\" (UniqueName: \"kubernetes.io/projected/218bc458-4d44-453a-84c8-2b7fe211580e-kube-api-access-pjs26\") pod \"llmisvc-controller-manager-6466c554d8-54d5w\" (UID: \"218bc458-4d44-453a-84c8-2b7fe211580e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-54d5w" Nov 03 14:18:44.086034 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:18:44.085940 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/218bc458-4d44-453a-84c8-2b7fe211580e-cert\") pod \"llmisvc-controller-manager-6466c554d8-54d5w\" (UID: \"218bc458-4d44-453a-84c8-2b7fe211580e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-54d5w" Nov 03 14:18:44.186621 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:18:44.186585 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjs26\" (UniqueName: \"kubernetes.io/projected/218bc458-4d44-453a-84c8-2b7fe211580e-kube-api-access-pjs26\") pod \"llmisvc-controller-manager-6466c554d8-54d5w\" (UID: \"218bc458-4d44-453a-84c8-2b7fe211580e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-54d5w" Nov 03 14:18:44.186810 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:18:44.186655 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/218bc458-4d44-453a-84c8-2b7fe211580e-cert\") pod \"llmisvc-controller-manager-6466c554d8-54d5w\" (UID: \"218bc458-4d44-453a-84c8-2b7fe211580e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-54d5w" Nov 03 14:18:44.189121 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:18:44.189093 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/218bc458-4d44-453a-84c8-2b7fe211580e-cert\") pod \"llmisvc-controller-manager-6466c554d8-54d5w\" (UID: \"218bc458-4d44-453a-84c8-2b7fe211580e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-54d5w" Nov 03 14:18:44.208776 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:18:44.208752 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjs26\" (UniqueName: \"kubernetes.io/projected/218bc458-4d44-453a-84c8-2b7fe211580e-kube-api-access-pjs26\") pod \"llmisvc-controller-manager-6466c554d8-54d5w\" (UID: \"218bc458-4d44-453a-84c8-2b7fe211580e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-54d5w" Nov 03 14:18:44.303935 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:18:44.303904 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-54d5w" Nov 03 14:18:44.599514 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:18:44.599491 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-54d5w"] Nov 03 14:18:44.601558 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:18:44.601523 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod218bc458_4d44_453a_84c8_2b7fe211580e.slice/crio-92a47428ec0c75a0676ed9e754a1bf9baa322c21e2f19b4a6c2d177705b9f897 WatchSource:0}: Error finding container 92a47428ec0c75a0676ed9e754a1bf9baa322c21e2f19b4a6c2d177705b9f897: Status 404 returned error can't find the container with id 92a47428ec0c75a0676ed9e754a1bf9baa322c21e2f19b4a6c2d177705b9f897 Nov 03 14:18:45.531767 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:18:45.531732 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-54d5w" event={"ID":"218bc458-4d44-453a-84c8-2b7fe211580e","Type":"ContainerStarted","Data":"34e192531fb5feb1b6f39686181a39e490fd29554b41d9d955eb452c77a1af6d"} Nov 03 14:18:45.532244 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:18:45.531790 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-54d5w" event={"ID":"218bc458-4d44-453a-84c8-2b7fe211580e","Type":"ContainerStarted","Data":"92a47428ec0c75a0676ed9e754a1bf9baa322c21e2f19b4a6c2d177705b9f897"} Nov 03 14:18:45.532244 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:18:45.531826 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-54d5w" Nov 03 14:18:45.576612 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:18:45.576561 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-54d5w" podStartSLOduration=2.212656042 podStartE2EDuration="2.576547489s" totalImagesPullingTime="363.891447ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:18:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:18:45.575356953 +0000 UTC m=+459.464146534" watchObservedRunningTime="2026-11-03 14:18:45.576547489 +0000 UTC m=+459.465337124" Nov 03 14:19:09.439746 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:19:09.439707 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:19:16.538901 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:19:16.538872 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-54d5w" Nov 03 14:19:16.620916 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:19:16.620872 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-75cwz"] Nov 03 14:19:16.621214 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:19:16.621181 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-75cwz" podUID="d887bc8f-e13f-4853-b079-41e503286e1a" containerName="manager" containerID="cri-o://22c8f3a146e990afcb1aff942a1b3ca4f9a004c3170a384f3ad1189ee58ed5a0" gracePeriod=30 Nov 03 14:19:16.875098 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:19:16.875068 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-75cwz" Nov 03 14:19:16.972668 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:19:16.972638 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d887bc8f-e13f-4853-b079-41e503286e1a-kube-api-access-gzdjq\" (UniqueName: \"kubernetes.io/projected/d887bc8f-e13f-4853-b079-41e503286e1a-kube-api-access-gzdjq\") pod \"d887bc8f-e13f-4853-b079-41e503286e1a\" (UID: \"d887bc8f-e13f-4853-b079-41e503286e1a\") " Nov 03 14:19:16.972858 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:19:16.972684 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d887bc8f-e13f-4853-b079-41e503286e1a-cert\" (UniqueName: \"kubernetes.io/secret/d887bc8f-e13f-4853-b079-41e503286e1a-cert\") pod \"d887bc8f-e13f-4853-b079-41e503286e1a\" (UID: \"d887bc8f-e13f-4853-b079-41e503286e1a\") " Nov 03 14:19:16.974972 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:19:16.974931 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d887bc8f-e13f-4853-b079-41e503286e1a-cert" pod "d887bc8f-e13f-4853-b079-41e503286e1a" (UID: "d887bc8f-e13f-4853-b079-41e503286e1a"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:19:16.975088 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:19:16.974997 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d887bc8f-e13f-4853-b079-41e503286e1a-kube-api-access-gzdjq" pod "d887bc8f-e13f-4853-b079-41e503286e1a" (UID: "d887bc8f-e13f-4853-b079-41e503286e1a"). InnerVolumeSpecName "kube-api-access-gzdjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:19:17.073901 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:19:17.073867 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzdjq\" (UniqueName: \"kubernetes.io/projected/d887bc8f-e13f-4853-b079-41e503286e1a-kube-api-access-gzdjq\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:19:17.073901 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:19:17.073894 5060 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d887bc8f-e13f-4853-b079-41e503286e1a-cert\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:19:17.696737 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:19:17.696652 5060 generic.go:356] "Generic (PLEG): container finished" podID="d887bc8f-e13f-4853-b079-41e503286e1a" containerID="22c8f3a146e990afcb1aff942a1b3ca4f9a004c3170a384f3ad1189ee58ed5a0" exitCode=0 Nov 03 14:19:17.696737 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:19:17.696705 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-75cwz" Nov 03 14:19:17.697275 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:19:17.696737 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-75cwz" event={"ID":"d887bc8f-e13f-4853-b079-41e503286e1a","Type":"ContainerDied","Data":"22c8f3a146e990afcb1aff942a1b3ca4f9a004c3170a384f3ad1189ee58ed5a0"} Nov 03 14:19:17.697275 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:19:17.696793 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-75cwz" event={"ID":"d887bc8f-e13f-4853-b079-41e503286e1a","Type":"ContainerDied","Data":"a1152f64802d266054f9e20fb194d5fe21e0d72c8d9e0cc6659d1dd5fa646ca4"} Nov 03 14:19:17.697275 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:19:17.696818 5060 scope.go:122] "RemoveContainer" containerID="22c8f3a146e990afcb1aff942a1b3ca4f9a004c3170a384f3ad1189ee58ed5a0" Nov 03 14:19:17.705983 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:19:17.705965 5060 scope.go:122] "RemoveContainer" containerID="22c8f3a146e990afcb1aff942a1b3ca4f9a004c3170a384f3ad1189ee58ed5a0" Nov 03 14:19:17.706356 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:19:17.706331 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22c8f3a146e990afcb1aff942a1b3ca4f9a004c3170a384f3ad1189ee58ed5a0\": container with ID starting with 22c8f3a146e990afcb1aff942a1b3ca4f9a004c3170a384f3ad1189ee58ed5a0 not found: ID does not exist" containerID="22c8f3a146e990afcb1aff942a1b3ca4f9a004c3170a384f3ad1189ee58ed5a0" Nov 03 14:19:17.706422 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:19:17.706367 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22c8f3a146e990afcb1aff942a1b3ca4f9a004c3170a384f3ad1189ee58ed5a0"} err="failed to get container status \"22c8f3a146e990afcb1aff942a1b3ca4f9a004c3170a384f3ad1189ee58ed5a0\": rpc error: code = NotFound desc = could not find container \"22c8f3a146e990afcb1aff942a1b3ca4f9a004c3170a384f3ad1189ee58ed5a0\": container with ID starting with 22c8f3a146e990afcb1aff942a1b3ca4f9a004c3170a384f3ad1189ee58ed5a0 not found: ID does not exist" Nov 03 14:19:17.752371 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:19:17.752333 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-75cwz"] Nov 03 14:19:17.765114 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:19:17.765080 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-75cwz"] Nov 03 14:19:18.248705 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:19:18.248664 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d887bc8f-e13f-4853-b079-41e503286e1a" path="/var/lib/kubelet/pods/d887bc8f-e13f-4853-b079-41e503286e1a/volumes" Nov 03 14:19:39.438465 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:19:39.438422 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:20:09.440136 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:20:09.440095 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:20:33.451942 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:33.451907 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs"] Nov 03 14:20:33.454616 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:33.452319 5060 cpu_manager.go:414] "containerMap: removing container" podUID="d887bc8f-e13f-4853-b079-41e503286e1a" containerName="manager" Nov 03 14:20:33.454616 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:33.452330 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d887bc8f-e13f-4853-b079-41e503286e1a" containerName="manager" Nov 03 14:20:33.454616 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:33.452411 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d887bc8f-e13f-4853-b079-41e503286e1a" containerName="manager" Nov 03 14:20:33.455569 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:33.455548 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" Nov 03 14:20:33.476935 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:33.476909 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 14:20:33.491965 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:33.491934 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs"] Nov 03 14:20:33.504022 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:33.503998 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4nspm\"" Nov 03 14:20:33.511302 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:33.511270 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44c38065-cf4a-47fd-a80f-9784767ae3ea-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7lxxs\" (UID: \"44c38065-cf4a-47fd-a80f-9784767ae3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" Nov 03 14:20:33.511302 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:33.511303 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lln5r\" (UniqueName: \"kubernetes.io/projected/44c38065-cf4a-47fd-a80f-9784767ae3ea-kube-api-access-lln5r\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7lxxs\" (UID: \"44c38065-cf4a-47fd-a80f-9784767ae3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" Nov 03 14:20:33.511494 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:33.511340 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44c38065-cf4a-47fd-a80f-9784767ae3ea-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7lxxs\" (UID: \"44c38065-cf4a-47fd-a80f-9784767ae3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" Nov 03 14:20:33.511494 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:33.511426 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c38065-cf4a-47fd-a80f-9784767ae3ea-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7lxxs\" (UID: \"44c38065-cf4a-47fd-a80f-9784767ae3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" Nov 03 14:20:33.511494 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:33.511473 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44c38065-cf4a-47fd-a80f-9784767ae3ea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7lxxs\" (UID: \"44c38065-cf4a-47fd-a80f-9784767ae3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" Nov 03 14:20:33.511657 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:33.511527 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c38065-cf4a-47fd-a80f-9784767ae3ea-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7lxxs\" (UID: \"44c38065-cf4a-47fd-a80f-9784767ae3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" Nov 03 14:20:33.511657 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:33.511558 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44c38065-cf4a-47fd-a80f-9784767ae3ea-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7lxxs\" (UID: \"44c38065-cf4a-47fd-a80f-9784767ae3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" Nov 03 14:20:33.511657 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:33.511578 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44c38065-cf4a-47fd-a80f-9784767ae3ea-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7lxxs\" (UID: \"44c38065-cf4a-47fd-a80f-9784767ae3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" Nov 03 14:20:33.511657 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:33.511593 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44c38065-cf4a-47fd-a80f-9784767ae3ea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7lxxs\" (UID: \"44c38065-cf4a-47fd-a80f-9784767ae3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" Nov 03 14:20:33.612606 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:33.612568 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44c38065-cf4a-47fd-a80f-9784767ae3ea-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7lxxs\" (UID: \"44c38065-cf4a-47fd-a80f-9784767ae3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" Nov 03 14:20:33.612606 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:33.612608 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c38065-cf4a-47fd-a80f-9784767ae3ea-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7lxxs\" (UID: \"44c38065-cf4a-47fd-a80f-9784767ae3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" Nov 03 14:20:33.612848 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:33.612650 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44c38065-cf4a-47fd-a80f-9784767ae3ea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7lxxs\" (UID: \"44c38065-cf4a-47fd-a80f-9784767ae3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" Nov 03 14:20:33.612848 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:33.612671 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c38065-cf4a-47fd-a80f-9784767ae3ea-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7lxxs\" (UID: \"44c38065-cf4a-47fd-a80f-9784767ae3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" Nov 03 14:20:33.612848 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:33.612700 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44c38065-cf4a-47fd-a80f-9784767ae3ea-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7lxxs\" (UID: \"44c38065-cf4a-47fd-a80f-9784767ae3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" Nov 03 14:20:33.612848 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:33.612732 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44c38065-cf4a-47fd-a80f-9784767ae3ea-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7lxxs\" (UID: \"44c38065-cf4a-47fd-a80f-9784767ae3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" Nov 03 14:20:33.613069 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:33.612894 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44c38065-cf4a-47fd-a80f-9784767ae3ea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7lxxs\" (UID: \"44c38065-cf4a-47fd-a80f-9784767ae3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" Nov 03 14:20:33.613069 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:33.612958 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44c38065-cf4a-47fd-a80f-9784767ae3ea-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7lxxs\" (UID: \"44c38065-cf4a-47fd-a80f-9784767ae3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" Nov 03 14:20:33.613069 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:33.612993 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lln5r\" (UniqueName: \"kubernetes.io/projected/44c38065-cf4a-47fd-a80f-9784767ae3ea-kube-api-access-lln5r\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7lxxs\" (UID: \"44c38065-cf4a-47fd-a80f-9784767ae3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" Nov 03 14:20:33.613216 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:33.613123 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c38065-cf4a-47fd-a80f-9784767ae3ea-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7lxxs\" (UID: \"44c38065-cf4a-47fd-a80f-9784767ae3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" Nov 03 14:20:33.613308 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:33.613279 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c38065-cf4a-47fd-a80f-9784767ae3ea-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7lxxs\" (UID: \"44c38065-cf4a-47fd-a80f-9784767ae3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" Nov 03 14:20:33.613456 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:33.613435 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44c38065-cf4a-47fd-a80f-9784767ae3ea-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7lxxs\" (UID: \"44c38065-cf4a-47fd-a80f-9784767ae3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" Nov 03 14:20:33.613620 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:33.613592 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44c38065-cf4a-47fd-a80f-9784767ae3ea-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7lxxs\" (UID: \"44c38065-cf4a-47fd-a80f-9784767ae3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" Nov 03 14:20:33.613767 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:33.613748 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44c38065-cf4a-47fd-a80f-9784767ae3ea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7lxxs\" (UID: \"44c38065-cf4a-47fd-a80f-9784767ae3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" Nov 03 14:20:33.615024 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:33.614999 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44c38065-cf4a-47fd-a80f-9784767ae3ea-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7lxxs\" (UID: \"44c38065-cf4a-47fd-a80f-9784767ae3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" Nov 03 14:20:33.615612 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:33.615593 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44c38065-cf4a-47fd-a80f-9784767ae3ea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7lxxs\" (UID: \"44c38065-cf4a-47fd-a80f-9784767ae3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" Nov 03 14:20:33.634282 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:33.634253 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44c38065-cf4a-47fd-a80f-9784767ae3ea-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7lxxs\" (UID: \"44c38065-cf4a-47fd-a80f-9784767ae3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" Nov 03 14:20:33.634534 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:33.634516 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lln5r\" (UniqueName: \"kubernetes.io/projected/44c38065-cf4a-47fd-a80f-9784767ae3ea-kube-api-access-lln5r\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7lxxs\" (UID: \"44c38065-cf4a-47fd-a80f-9784767ae3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" Nov 03 14:20:33.770604 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:33.770566 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" Nov 03 14:20:34.081635 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:34.081604 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs"] Nov 03 14:20:34.084212 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:20:34.084183 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44c38065_cf4a_47fd_a80f_9784767ae3ea.slice/crio-3ac3ca6d84bd068b85541a1da6341ecbc89ec0b2bc65d2e0263732ba35169e7a WatchSource:0}: Error finding container 3ac3ca6d84bd068b85541a1da6341ecbc89ec0b2bc65d2e0263732ba35169e7a: Status 404 returned error can't find the container with id 3ac3ca6d84bd068b85541a1da6341ecbc89ec0b2bc65d2e0263732ba35169e7a Nov 03 14:20:35.091725 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:35.091679 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" event={"ID":"44c38065-cf4a-47fd-a80f-9784767ae3ea","Type":"ContainerStarted","Data":"3ac3ca6d84bd068b85541a1da6341ecbc89ec0b2bc65d2e0263732ba35169e7a"} Nov 03 14:20:36.688085 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:36.688053 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:20:36.688372 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:36.688122 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:20:36.688372 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:36.688154 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:20:37.110796 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:37.109765 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" event={"ID":"44c38065-cf4a-47fd-a80f-9784767ae3ea","Type":"ContainerStarted","Data":"3b816520e9f4520b8feb02852c83b1fa3c11327e768a0a39b97bc0b347b3a533"} Nov 03 14:20:37.171411 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:37.171356 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" podStartSLOduration=1.570118825 podStartE2EDuration="4.171338459s" totalImagesPullingTime="2.601219634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:20:37.170102552 +0000 UTC m=+571.058892133" watchObservedRunningTime="2026-11-03 14:20:37.171338459 +0000 UTC m=+571.060128039" Nov 03 14:20:37.771629 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:37.771592 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" Nov 03 14:20:37.773265 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:37.773239 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 03 14:20:37.773389 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:37.773285 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" podUID="44c38065-cf4a-47fd-a80f-9784767ae3ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 03 14:20:38.771474 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:38.771437 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 03 14:20:38.771673 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:38.771497 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" podUID="44c38065-cf4a-47fd-a80f-9784767ae3ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 03 14:20:39.445399 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:20:39.445359 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:20:39.771788 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:39.771756 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 03 14:20:39.772184 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:39.771809 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" podUID="44c38065-cf4a-47fd-a80f-9784767ae3ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 03 14:20:40.771019 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:40.770981 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 03 14:20:40.771201 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:40.771033 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" podUID="44c38065-cf4a-47fd-a80f-9784767ae3ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 03 14:20:41.771991 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:41.771951 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 03 14:20:41.772463 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:41.772009 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" podUID="44c38065-cf4a-47fd-a80f-9784767ae3ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 03 14:20:42.770949 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:42.770914 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 03 14:20:42.771118 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:42.770978 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" podUID="44c38065-cf4a-47fd-a80f-9784767ae3ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 03 14:20:43.771264 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:43.771215 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 03 14:20:43.771662 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:43.771289 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" podUID="44c38065-cf4a-47fd-a80f-9784767ae3ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 03 14:20:44.771440 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:44.771396 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 03 14:20:44.771948 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:44.771458 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" podUID="44c38065-cf4a-47fd-a80f-9784767ae3ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 03 14:20:45.882913 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:45.882878 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" Nov 03 14:20:45.883993 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:45.883967 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" Nov 03 14:20:45.884474 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:45.884454 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" Nov 03 14:20:48.540960 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:48.540879 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg"] Nov 03 14:20:48.544495 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:48.544469 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" Nov 03 14:20:48.570921 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:48.570890 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg"] Nov 03 14:20:48.632533 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:48.632493 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lpzqg\" (UID: \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" Nov 03 14:20:48.632727 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:48.632546 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lpzqg\" (UID: \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" Nov 03 14:20:48.632727 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:48.632630 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lpzqg\" (UID: \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" Nov 03 14:20:48.632727 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:48.632716 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lpzqg\" (UID: \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" Nov 03 14:20:48.632932 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:48.632809 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lpzqg\" (UID: \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" Nov 03 14:20:48.632932 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:48.632880 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6fnm\" (UniqueName: \"kubernetes.io/projected/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-kube-api-access-c6fnm\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lpzqg\" (UID: \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" Nov 03 14:20:48.633038 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:48.632950 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lpzqg\" (UID: \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" Nov 03 14:20:48.633038 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:48.633000 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lpzqg\" (UID: \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" Nov 03 14:20:48.633133 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:48.633036 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lpzqg\" (UID: \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" Nov 03 14:20:48.733879 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:48.733815 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lpzqg\" (UID: \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" Nov 03 14:20:48.734081 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:48.733892 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lpzqg\" (UID: \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" Nov 03 14:20:48.734081 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:48.733911 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lpzqg\" (UID: \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" Nov 03 14:20:48.734081 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:48.733926 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lpzqg\" (UID: \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" Nov 03 14:20:48.734081 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:48.733944 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lpzqg\" (UID: \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" Nov 03 14:20:48.734081 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:48.733962 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6fnm\" (UniqueName: \"kubernetes.io/projected/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-kube-api-access-c6fnm\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lpzqg\" (UID: \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" Nov 03 14:20:48.734356 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:48.734317 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lpzqg\" (UID: \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" Nov 03 14:20:48.734418 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:48.734373 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lpzqg\" (UID: \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" Nov 03 14:20:48.734418 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:48.734390 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lpzqg\" (UID: \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" Nov 03 14:20:48.734520 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:48.734432 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lpzqg\" (UID: \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" Nov 03 14:20:48.734520 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:48.734466 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lpzqg\" (UID: \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" Nov 03 14:20:48.734629 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:48.734607 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lpzqg\" (UID: \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" Nov 03 14:20:48.734815 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:48.734794 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lpzqg\" (UID: \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" Nov 03 14:20:48.734920 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:48.734885 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lpzqg\" (UID: \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" Nov 03 14:20:48.736928 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:48.736907 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lpzqg\" (UID: \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" Nov 03 14:20:48.737048 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:48.737025 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lpzqg\" (UID: \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" Nov 03 14:20:48.756730 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:48.756707 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6fnm\" (UniqueName: \"kubernetes.io/projected/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-kube-api-access-c6fnm\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lpzqg\" (UID: \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" Nov 03 14:20:48.757425 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:48.757397 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lpzqg\" (UID: \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" Nov 03 14:20:48.859965 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:48.859869 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" Nov 03 14:20:49.173509 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:49.173364 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg"] Nov 03 14:20:49.175683 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:20:49.175647 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76bc22bf_ed6f_4f6b_a225_5611aa3b73b0.slice/crio-73d985cf26b3be037dc028071e0b31167938d3d999798ef053f8eb28b332d880 WatchSource:0}: Error finding container 73d985cf26b3be037dc028071e0b31167938d3d999798ef053f8eb28b332d880: Status 404 returned error can't find the container with id 73d985cf26b3be037dc028071e0b31167938d3d999798ef053f8eb28b332d880 Nov 03 14:20:49.178977 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:49.178944 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:20:49.179058 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:49.179012 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:20:49.179058 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:49.179041 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:20:50.186715 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:50.186674 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" event={"ID":"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0","Type":"ContainerStarted","Data":"84208702fd3a29e1d27ad3e868908ab06f7b863861cf91399568acab9f972c7f"} Nov 03 14:20:50.187140 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:50.186739 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" event={"ID":"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0","Type":"ContainerStarted","Data":"73d985cf26b3be037dc028071e0b31167938d3d999798ef053f8eb28b332d880"} Nov 03 14:20:50.276061 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:50.276005 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" podStartSLOduration=2.275991583 podStartE2EDuration="2.275991583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:20:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:20:50.262918064 +0000 UTC m=+584.151707645" watchObservedRunningTime="2026-11-03 14:20:50.275991583 +0000 UTC m=+584.164781163" Nov 03 14:20:50.860562 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:50.860528 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" Nov 03 14:20:50.862142 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:50.862112 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 03 14:20:50.862272 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:50.862161 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" podUID="76bc22bf-ed6f-4f6b-a225-5611aa3b73b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 03 14:20:51.860572 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:51.860534 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 03 14:20:51.861062 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:51.860597 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" podUID="76bc22bf-ed6f-4f6b-a225-5611aa3b73b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 03 14:20:52.861046 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:52.861010 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 03 14:20:52.861433 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:52.861068 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" podUID="76bc22bf-ed6f-4f6b-a225-5611aa3b73b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 03 14:20:53.861011 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:53.860975 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 03 14:20:53.861200 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:53.861030 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" podUID="76bc22bf-ed6f-4f6b-a225-5611aa3b73b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 03 14:20:54.861251 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:54.861201 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 03 14:20:54.861743 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:54.861265 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" podUID="76bc22bf-ed6f-4f6b-a225-5611aa3b73b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 03 14:20:55.861190 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:55.861147 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 03 14:20:55.861373 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:55.861230 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" podUID="76bc22bf-ed6f-4f6b-a225-5611aa3b73b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 03 14:20:56.860799 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:56.860756 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 03 14:20:56.861015 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:56.860867 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" podUID="76bc22bf-ed6f-4f6b-a225-5611aa3b73b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 03 14:20:58.646860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:58.646745 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" Nov 03 14:20:58.647310 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:58.647138 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" Nov 03 14:20:58.647973 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:20:58.647953 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" Nov 03 14:21:03.584507 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:03.584472 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg"] Nov 03 14:21:03.588433 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:03.588412 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" Nov 03 14:21:03.622167 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:03.622137 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg"] Nov 03 14:21:03.683180 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:03.683139 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bzkwg\" (UID: \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" Nov 03 14:21:03.683180 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:03.683178 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bzkwg\" (UID: \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" Nov 03 14:21:03.683410 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:03.683199 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bzkwg\" (UID: \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" Nov 03 14:21:03.683410 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:03.683369 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bzkwg\" (UID: \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" Nov 03 14:21:03.683410 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:03.683399 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bzkwg\" (UID: \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" Nov 03 14:21:03.683551 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:03.683424 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bzkwg\" (UID: \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" Nov 03 14:21:03.683599 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:03.683549 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdtlx\" (UniqueName: \"kubernetes.io/projected/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-kube-api-access-pdtlx\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bzkwg\" (UID: \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" Nov 03 14:21:03.683599 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:03.683584 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bzkwg\" (UID: \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" Nov 03 14:21:03.683697 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:03.683625 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bzkwg\" (UID: \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" Nov 03 14:21:03.784237 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:03.784200 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdtlx\" (UniqueName: \"kubernetes.io/projected/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-kube-api-access-pdtlx\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bzkwg\" (UID: \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" Nov 03 14:21:03.784237 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:03.784239 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bzkwg\" (UID: \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" Nov 03 14:21:03.784498 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:03.784264 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bzkwg\" (UID: \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" Nov 03 14:21:03.784498 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:03.784293 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bzkwg\" (UID: \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" Nov 03 14:21:03.784498 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:03.784318 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bzkwg\" (UID: \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" Nov 03 14:21:03.784498 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:03.784337 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bzkwg\" (UID: \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" Nov 03 14:21:03.784498 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:03.784382 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bzkwg\" (UID: \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" Nov 03 14:21:03.784498 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:03.784409 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bzkwg\" (UID: \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" Nov 03 14:21:03.784498 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:03.784431 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bzkwg\" (UID: \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" Nov 03 14:21:03.784933 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:03.784887 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bzkwg\" (UID: \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" Nov 03 14:21:03.785067 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:03.784930 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bzkwg\" (UID: \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" Nov 03 14:21:03.785067 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:03.784973 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bzkwg\" (UID: \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" Nov 03 14:21:03.785163 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:03.785092 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bzkwg\" (UID: \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" Nov 03 14:21:03.785163 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:03.785099 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bzkwg\" (UID: \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" Nov 03 14:21:03.786658 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:03.786634 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bzkwg\" (UID: \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" Nov 03 14:21:03.786916 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:03.786895 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bzkwg\" (UID: \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" Nov 03 14:21:03.806190 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:03.806164 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdtlx\" (UniqueName: \"kubernetes.io/projected/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-kube-api-access-pdtlx\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bzkwg\" (UID: \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" Nov 03 14:21:03.806281 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:03.806211 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bzkwg\" (UID: \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" Nov 03 14:21:03.901932 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:03.901827 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" Nov 03 14:21:04.216987 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:04.216956 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg"] Nov 03 14:21:04.218829 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:21:04.218798 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9737ed5d_be9c_45ef_afd6_977eddcc3ce6.slice/crio-9a4550eb023faf820f9f50bba0a68f1cc15bdb402e465bc0459df2c1afa64913 WatchSource:0}: Error finding container 9a4550eb023faf820f9f50bba0a68f1cc15bdb402e465bc0459df2c1afa64913: Status 404 returned error can't find the container with id 9a4550eb023faf820f9f50bba0a68f1cc15bdb402e465bc0459df2c1afa64913 Nov 03 14:21:04.221935 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:04.221891 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:21:04.222041 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:04.221976 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:21:04.222041 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:04.222020 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:21:04.288909 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:04.288874 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" event={"ID":"9737ed5d-be9c-45ef-afd6-977eddcc3ce6","Type":"ContainerStarted","Data":"6268b68cb5af006ac4996e17d774cc0536c0536bcdfd1bfc5eb67ed7fc2f46a6"} Nov 03 14:21:04.289027 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:04.288939 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" event={"ID":"9737ed5d-be9c-45ef-afd6-977eddcc3ce6","Type":"ContainerStarted","Data":"9a4550eb023faf820f9f50bba0a68f1cc15bdb402e465bc0459df2c1afa64913"} Nov 03 14:21:04.344912 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:04.342074 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" podStartSLOduration=1.342053469 podStartE2EDuration="1.342053469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:21:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:21:04.335250572 +0000 UTC m=+598.224040155" watchObservedRunningTime="2026-11-03 14:21:04.342053469 +0000 UTC m=+598.230843051" Nov 03 14:21:04.902776 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:04.902740 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" Nov 03 14:21:05.910866 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:05.910594 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 03 14:21:05.910866 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:05.910661 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" podUID="9737ed5d-be9c-45ef-afd6-977eddcc3ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 03 14:21:06.340126 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:06.340095 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/3.log" Nov 03 14:21:06.340984 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:06.340961 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/3.log" Nov 03 14:21:06.905388 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:06.905351 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 03 14:21:06.905694 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:06.905650 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" podUID="9737ed5d-be9c-45ef-afd6-977eddcc3ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 03 14:21:07.908887 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:07.903264 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 03 14:21:07.908887 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:07.903331 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" podUID="9737ed5d-be9c-45ef-afd6-977eddcc3ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 03 14:21:08.904859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:08.902250 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 03 14:21:08.904859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:08.902330 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" podUID="9737ed5d-be9c-45ef-afd6-977eddcc3ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 03 14:21:09.445618 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:21:09.445584 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:21:09.903251 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:09.903209 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 03 14:21:09.903561 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:09.903530 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" podUID="9737ed5d-be9c-45ef-afd6-977eddcc3ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 03 14:21:10.902436 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:10.902396 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 03 14:21:10.902985 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:10.902485 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" podUID="9737ed5d-be9c-45ef-afd6-977eddcc3ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 03 14:21:11.902653 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:11.902614 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 03 14:21:11.903072 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:11.902683 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" podUID="9737ed5d-be9c-45ef-afd6-977eddcc3ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 03 14:21:13.561123 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:13.561090 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" Nov 03 14:21:13.561547 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:13.561395 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" Nov 03 14:21:13.562267 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:13.562250 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" Nov 03 14:21:39.440301 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:21:39.440265 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:21:44.367056 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:44.367012 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd"] Nov 03 14:21:44.372059 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:44.372031 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" Nov 03 14:21:44.394429 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:44.394405 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-1ffaa0-openshift-default-dockercfg-k7w6w\"" Nov 03 14:21:44.407965 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:44.407934 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd"] Nov 03 14:21:44.422509 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:44.422465 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8d7e93-3520-4a18-916a-ac77ae09f885-credential-socket\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd\" (UID: \"6c8d7e93-3520-4a18-916a-ac77ae09f885\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" Nov 03 14:21:44.422509 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:44.422496 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c8d7e93-3520-4a18-916a-ac77ae09f885-istio-envoy\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd\" (UID: \"6c8d7e93-3520-4a18-916a-ac77ae09f885\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" Nov 03 14:21:44.422724 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:44.422607 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8d7e93-3520-4a18-916a-ac77ae09f885-workload-socket\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd\" (UID: \"6c8d7e93-3520-4a18-916a-ac77ae09f885\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" Nov 03 14:21:44.422724 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:44.422635 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c8d7e93-3520-4a18-916a-ac77ae09f885-workload-certs\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd\" (UID: \"6c8d7e93-3520-4a18-916a-ac77ae09f885\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" Nov 03 14:21:44.422724 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:44.422659 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c8d7e93-3520-4a18-916a-ac77ae09f885-istio-data\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd\" (UID: \"6c8d7e93-3520-4a18-916a-ac77ae09f885\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" Nov 03 14:21:44.422724 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:44.422684 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c8d7e93-3520-4a18-916a-ac77ae09f885-istio-token\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd\" (UID: \"6c8d7e93-3520-4a18-916a-ac77ae09f885\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" Nov 03 14:21:44.422939 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:44.422764 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c8d7e93-3520-4a18-916a-ac77ae09f885-istio-podinfo\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd\" (UID: \"6c8d7e93-3520-4a18-916a-ac77ae09f885\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" Nov 03 14:21:44.422939 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:44.422815 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmc62\" (UniqueName: \"kubernetes.io/projected/6c8d7e93-3520-4a18-916a-ac77ae09f885-kube-api-access-gmc62\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd\" (UID: \"6c8d7e93-3520-4a18-916a-ac77ae09f885\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" Nov 03 14:21:44.422939 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:44.422917 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c8d7e93-3520-4a18-916a-ac77ae09f885-istiod-ca-cert\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd\" (UID: \"6c8d7e93-3520-4a18-916a-ac77ae09f885\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" Nov 03 14:21:44.523876 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:44.523816 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmc62\" (UniqueName: \"kubernetes.io/projected/6c8d7e93-3520-4a18-916a-ac77ae09f885-kube-api-access-gmc62\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd\" (UID: \"6c8d7e93-3520-4a18-916a-ac77ae09f885\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" Nov 03 14:21:44.524060 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:44.523905 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c8d7e93-3520-4a18-916a-ac77ae09f885-istiod-ca-cert\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd\" (UID: \"6c8d7e93-3520-4a18-916a-ac77ae09f885\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" Nov 03 14:21:44.524060 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:44.523961 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8d7e93-3520-4a18-916a-ac77ae09f885-credential-socket\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd\" (UID: \"6c8d7e93-3520-4a18-916a-ac77ae09f885\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" Nov 03 14:21:44.524060 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:44.523985 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c8d7e93-3520-4a18-916a-ac77ae09f885-istio-envoy\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd\" (UID: \"6c8d7e93-3520-4a18-916a-ac77ae09f885\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" Nov 03 14:21:44.524060 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:44.524028 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8d7e93-3520-4a18-916a-ac77ae09f885-workload-socket\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd\" (UID: \"6c8d7e93-3520-4a18-916a-ac77ae09f885\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" Nov 03 14:21:44.524060 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:44.524055 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c8d7e93-3520-4a18-916a-ac77ae09f885-workload-certs\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd\" (UID: \"6c8d7e93-3520-4a18-916a-ac77ae09f885\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" Nov 03 14:21:44.524339 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:44.524078 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c8d7e93-3520-4a18-916a-ac77ae09f885-istio-data\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd\" (UID: \"6c8d7e93-3520-4a18-916a-ac77ae09f885\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" Nov 03 14:21:44.524339 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:44.524114 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c8d7e93-3520-4a18-916a-ac77ae09f885-istio-token\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd\" (UID: \"6c8d7e93-3520-4a18-916a-ac77ae09f885\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" Nov 03 14:21:44.524339 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:44.524169 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c8d7e93-3520-4a18-916a-ac77ae09f885-istio-podinfo\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd\" (UID: \"6c8d7e93-3520-4a18-916a-ac77ae09f885\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" Nov 03 14:21:44.524494 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:44.524414 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8d7e93-3520-4a18-916a-ac77ae09f885-workload-socket\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd\" (UID: \"6c8d7e93-3520-4a18-916a-ac77ae09f885\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" Nov 03 14:21:44.524621 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:44.524601 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8d7e93-3520-4a18-916a-ac77ae09f885-credential-socket\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd\" (UID: \"6c8d7e93-3520-4a18-916a-ac77ae09f885\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" Nov 03 14:21:44.524742 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:44.524718 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c8d7e93-3520-4a18-916a-ac77ae09f885-istiod-ca-cert\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd\" (UID: \"6c8d7e93-3520-4a18-916a-ac77ae09f885\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" Nov 03 14:21:44.524883 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:44.524827 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c8d7e93-3520-4a18-916a-ac77ae09f885-istio-data\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd\" (UID: \"6c8d7e93-3520-4a18-916a-ac77ae09f885\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" Nov 03 14:21:44.525052 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:44.525031 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c8d7e93-3520-4a18-916a-ac77ae09f885-workload-certs\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd\" (UID: \"6c8d7e93-3520-4a18-916a-ac77ae09f885\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" Nov 03 14:21:44.526991 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:44.526959 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c8d7e93-3520-4a18-916a-ac77ae09f885-istio-envoy\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd\" (UID: \"6c8d7e93-3520-4a18-916a-ac77ae09f885\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" Nov 03 14:21:44.527122 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:44.527051 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c8d7e93-3520-4a18-916a-ac77ae09f885-istio-podinfo\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd\" (UID: \"6c8d7e93-3520-4a18-916a-ac77ae09f885\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" Nov 03 14:21:44.546764 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:44.546732 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmc62\" (UniqueName: \"kubernetes.io/projected/6c8d7e93-3520-4a18-916a-ac77ae09f885-kube-api-access-gmc62\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd\" (UID: \"6c8d7e93-3520-4a18-916a-ac77ae09f885\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" Nov 03 14:21:44.546764 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:44.546742 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c8d7e93-3520-4a18-916a-ac77ae09f885-istio-token\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd\" (UID: \"6c8d7e93-3520-4a18-916a-ac77ae09f885\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" Nov 03 14:21:44.692091 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:44.691992 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" Nov 03 14:21:45.022487 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:45.022451 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd"] Nov 03 14:21:45.023951 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:21:45.023914 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c8d7e93_3520_4a18_916a_ac77ae09f885.slice/crio-a6d3d89160dbd24b8c929373bddba0afe3be9d4cdc04ba27e58c8530285daba8 WatchSource:0}: Error finding container a6d3d89160dbd24b8c929373bddba0afe3be9d4cdc04ba27e58c8530285daba8: Status 404 returned error can't find the container with id a6d3d89160dbd24b8c929373bddba0afe3be9d4cdc04ba27e58c8530285daba8 Nov 03 14:21:45.026811 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:45.026772 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:21:45.026960 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:45.026936 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:21:45.027012 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:45.026996 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:21:45.593223 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:45.593186 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" event={"ID":"6c8d7e93-3520-4a18-916a-ac77ae09f885","Type":"ContainerStarted","Data":"572d43f258e5370b50d58feb6d259d6c0b88b376677bad0e8cc855d1e9381b29"} Nov 03 14:21:45.593733 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:45.593250 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" event={"ID":"6c8d7e93-3520-4a18-916a-ac77ae09f885","Type":"ContainerStarted","Data":"a6d3d89160dbd24b8c929373bddba0afe3be9d4cdc04ba27e58c8530285daba8"} Nov 03 14:21:45.665527 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:45.665464 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" podStartSLOduration=1.665446653 podStartE2EDuration="1.665446653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:21:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:21:45.651865656 +0000 UTC m=+639.540655238" watchObservedRunningTime="2026-11-03 14:21:45.665446653 +0000 UTC m=+639.554236234" Nov 03 14:21:45.692918 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:45.692892 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" Nov 03 14:21:46.709854 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:46.709171 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" Nov 03 14:21:47.624854 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:47.624618 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" Nov 03 14:21:47.635860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:47.635170 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" Nov 03 14:21:54.975813 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:54.975774 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz"] Nov 03 14:21:54.979945 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:54.979921 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" Nov 03 14:21:54.998556 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:54.998535 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-156ae1-openshift-default-dockercfg-hh5m9\"" Nov 03 14:21:55.013702 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:55.013674 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz"] Nov 03 14:21:55.070909 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:55.070879 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3327f34-66f3-4a70-915a-aa5ad8685fad-istiod-ca-cert\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz\" (UID: \"b3327f34-66f3-4a70-915a-aa5ad8685fad\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" Nov 03 14:21:55.071088 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:55.070926 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3327f34-66f3-4a70-915a-aa5ad8685fad-istio-envoy\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz\" (UID: \"b3327f34-66f3-4a70-915a-aa5ad8685fad\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" Nov 03 14:21:55.071088 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:55.070991 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3327f34-66f3-4a70-915a-aa5ad8685fad-credential-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz\" (UID: \"b3327f34-66f3-4a70-915a-aa5ad8685fad\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" Nov 03 14:21:55.071088 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:55.071065 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3327f34-66f3-4a70-915a-aa5ad8685fad-istio-podinfo\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz\" (UID: \"b3327f34-66f3-4a70-915a-aa5ad8685fad\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" Nov 03 14:21:55.071216 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:55.071100 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5kq5\" (UniqueName: \"kubernetes.io/projected/b3327f34-66f3-4a70-915a-aa5ad8685fad-kube-api-access-k5kq5\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz\" (UID: \"b3327f34-66f3-4a70-915a-aa5ad8685fad\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" Nov 03 14:21:55.071216 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:55.071129 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3327f34-66f3-4a70-915a-aa5ad8685fad-istio-token\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz\" (UID: \"b3327f34-66f3-4a70-915a-aa5ad8685fad\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" Nov 03 14:21:55.071216 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:55.071205 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3327f34-66f3-4a70-915a-aa5ad8685fad-istio-data\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz\" (UID: \"b3327f34-66f3-4a70-915a-aa5ad8685fad\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" Nov 03 14:21:55.071324 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:55.071242 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3327f34-66f3-4a70-915a-aa5ad8685fad-workload-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz\" (UID: \"b3327f34-66f3-4a70-915a-aa5ad8685fad\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" Nov 03 14:21:55.071324 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:55.071271 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3327f34-66f3-4a70-915a-aa5ad8685fad-workload-certs\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz\" (UID: \"b3327f34-66f3-4a70-915a-aa5ad8685fad\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" Nov 03 14:21:55.171934 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:55.171896 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3327f34-66f3-4a70-915a-aa5ad8685fad-workload-certs\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz\" (UID: \"b3327f34-66f3-4a70-915a-aa5ad8685fad\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" Nov 03 14:21:55.172124 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:55.171952 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3327f34-66f3-4a70-915a-aa5ad8685fad-istiod-ca-cert\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz\" (UID: \"b3327f34-66f3-4a70-915a-aa5ad8685fad\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" Nov 03 14:21:55.172124 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:55.171984 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3327f34-66f3-4a70-915a-aa5ad8685fad-istio-envoy\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz\" (UID: \"b3327f34-66f3-4a70-915a-aa5ad8685fad\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" Nov 03 14:21:55.172124 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:55.172007 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3327f34-66f3-4a70-915a-aa5ad8685fad-credential-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz\" (UID: \"b3327f34-66f3-4a70-915a-aa5ad8685fad\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" Nov 03 14:21:55.172124 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:55.172033 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3327f34-66f3-4a70-915a-aa5ad8685fad-istio-podinfo\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz\" (UID: \"b3327f34-66f3-4a70-915a-aa5ad8685fad\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" Nov 03 14:21:55.172124 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:55.172050 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5kq5\" (UniqueName: \"kubernetes.io/projected/b3327f34-66f3-4a70-915a-aa5ad8685fad-kube-api-access-k5kq5\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz\" (UID: \"b3327f34-66f3-4a70-915a-aa5ad8685fad\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" Nov 03 14:21:55.172124 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:55.172072 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3327f34-66f3-4a70-915a-aa5ad8685fad-istio-token\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz\" (UID: \"b3327f34-66f3-4a70-915a-aa5ad8685fad\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" Nov 03 14:21:55.172124 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:55.172110 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3327f34-66f3-4a70-915a-aa5ad8685fad-istio-data\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz\" (UID: \"b3327f34-66f3-4a70-915a-aa5ad8685fad\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" Nov 03 14:21:55.172496 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:55.172157 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3327f34-66f3-4a70-915a-aa5ad8685fad-workload-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz\" (UID: \"b3327f34-66f3-4a70-915a-aa5ad8685fad\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" Nov 03 14:21:55.172496 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:55.172369 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3327f34-66f3-4a70-915a-aa5ad8685fad-workload-certs\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz\" (UID: \"b3327f34-66f3-4a70-915a-aa5ad8685fad\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" Nov 03 14:21:55.172496 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:55.172427 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3327f34-66f3-4a70-915a-aa5ad8685fad-credential-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz\" (UID: \"b3327f34-66f3-4a70-915a-aa5ad8685fad\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" Nov 03 14:21:55.172649 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:55.172628 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3327f34-66f3-4a70-915a-aa5ad8685fad-workload-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz\" (UID: \"b3327f34-66f3-4a70-915a-aa5ad8685fad\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" Nov 03 14:21:55.172801 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:55.172778 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3327f34-66f3-4a70-915a-aa5ad8685fad-istio-data\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz\" (UID: \"b3327f34-66f3-4a70-915a-aa5ad8685fad\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" Nov 03 14:21:55.173074 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:55.173056 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3327f34-66f3-4a70-915a-aa5ad8685fad-istiod-ca-cert\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz\" (UID: \"b3327f34-66f3-4a70-915a-aa5ad8685fad\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" Nov 03 14:21:55.174269 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:55.174248 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3327f34-66f3-4a70-915a-aa5ad8685fad-istio-envoy\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz\" (UID: \"b3327f34-66f3-4a70-915a-aa5ad8685fad\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" Nov 03 14:21:55.174658 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:55.174636 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3327f34-66f3-4a70-915a-aa5ad8685fad-istio-podinfo\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz\" (UID: \"b3327f34-66f3-4a70-915a-aa5ad8685fad\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" Nov 03 14:21:55.194118 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:55.194087 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3327f34-66f3-4a70-915a-aa5ad8685fad-istio-token\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz\" (UID: \"b3327f34-66f3-4a70-915a-aa5ad8685fad\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" Nov 03 14:21:55.194213 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:55.194154 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5kq5\" (UniqueName: \"kubernetes.io/projected/b3327f34-66f3-4a70-915a-aa5ad8685fad-kube-api-access-k5kq5\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz\" (UID: \"b3327f34-66f3-4a70-915a-aa5ad8685fad\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" Nov 03 14:21:55.293527 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:55.293490 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" Nov 03 14:21:55.613954 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:55.613872 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz"] Nov 03 14:21:55.617804 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:21:55.617777 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3327f34_66f3_4a70_915a_aa5ad8685fad.slice/crio-ee6708b33ac72e956689651504c8ee13d989bd5274bfdd3d9f37391e0b320047 WatchSource:0}: Error finding container ee6708b33ac72e956689651504c8ee13d989bd5274bfdd3d9f37391e0b320047: Status 404 returned error can't find the container with id ee6708b33ac72e956689651504c8ee13d989bd5274bfdd3d9f37391e0b320047 Nov 03 14:21:55.620152 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:55.620118 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:21:55.620241 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:55.620185 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:21:55.620241 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:55.620237 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:21:55.688421 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:55.688389 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" event={"ID":"b3327f34-66f3-4a70-915a-aa5ad8685fad","Type":"ContainerStarted","Data":"42e4fcc0f5d87a398aeeabbebcddac52534e1612a7107b5f0f20c1d3b29b20a4"} Nov 03 14:21:55.688563 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:55.688448 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" event={"ID":"b3327f34-66f3-4a70-915a-aa5ad8685fad","Type":"ContainerStarted","Data":"ee6708b33ac72e956689651504c8ee13d989bd5274bfdd3d9f37391e0b320047"} Nov 03 14:21:55.736805 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:55.736741 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" podStartSLOduration=1.736722049 podStartE2EDuration="1.736722049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:21:55.733908618 +0000 UTC m=+649.622698202" watchObservedRunningTime="2026-11-03 14:21:55.736722049 +0000 UTC m=+649.625511632" Nov 03 14:21:56.294924 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:56.294883 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" Nov 03 14:21:56.298867 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:56.296319 5060 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 03 14:21:56.298867 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:56.296384 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" podUID="b3327f34-66f3-4a70-915a-aa5ad8685fad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 03 14:21:57.294875 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:57.294814 5060 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 03 14:21:57.295045 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:57.294896 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" podUID="b3327f34-66f3-4a70-915a-aa5ad8685fad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 03 14:21:58.293954 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:58.293914 5060 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 03 14:21:58.294149 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:58.293984 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" podUID="b3327f34-66f3-4a70-915a-aa5ad8685fad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 03 14:21:59.294671 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:59.294630 5060 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 03 14:21:59.295183 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:21:59.294697 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" podUID="b3327f34-66f3-4a70-915a-aa5ad8685fad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 03 14:22:00.294465 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:00.294433 5060 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 03 14:22:00.294644 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:00.294483 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" podUID="b3327f34-66f3-4a70-915a-aa5ad8685fad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 03 14:22:01.294773 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:01.294735 5060 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 03 14:22:01.295273 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:01.294790 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" podUID="b3327f34-66f3-4a70-915a-aa5ad8685fad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 03 14:22:02.551978 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:02.551941 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" Nov 03 14:22:02.552527 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:02.552500 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" Nov 03 14:22:02.553348 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:02.553332 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" Nov 03 14:22:03.464596 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:03.464563 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-4429aa/praxis-route-4429aa-kserve-668c498f79-4pb6w"] Nov 03 14:22:03.468633 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:03.468605 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-4429aa/praxis-route-4429aa-kserve-668c498f79-4pb6w" Nov 03 14:22:03.501369 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:03.501337 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-4429aa\"/\"default-dockercfg-kkggr\"" Nov 03 14:22:03.501546 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:03.501383 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-4429aa\"/\"openshift-service-ca.crt\"" Nov 03 14:22:03.501546 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:03.501429 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-4429aa\"/\"praxis-route-4429aa-kserve-self-signed-certs\"" Nov 03 14:22:03.501546 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:03.501471 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-4429aa\"/\"kube-root-ca.crt\"" Nov 03 14:22:03.505206 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:03.505180 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-4429aa/praxis-route-4429aa-kserve-668c498f79-4pb6w"] Nov 03 14:22:03.597513 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:03.597456 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13089e4b-aad1-4aad-a869-1ef46d93bd6e-dshm\") pod \"praxis-route-4429aa-kserve-668c498f79-4pb6w\" (UID: \"13089e4b-aad1-4aad-a869-1ef46d93bd6e\") " pod="ai-tenant-e2e-praxis-a-4429aa/praxis-route-4429aa-kserve-668c498f79-4pb6w" Nov 03 14:22:03.598046 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:03.597581 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13089e4b-aad1-4aad-a869-1ef46d93bd6e-tmp-dir\") pod \"praxis-route-4429aa-kserve-668c498f79-4pb6w\" (UID: \"13089e4b-aad1-4aad-a869-1ef46d93bd6e\") " pod="ai-tenant-e2e-praxis-a-4429aa/praxis-route-4429aa-kserve-668c498f79-4pb6w" Nov 03 14:22:03.598046 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:03.597625 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zxsl\" (UniqueName: \"kubernetes.io/projected/13089e4b-aad1-4aad-a869-1ef46d93bd6e-kube-api-access-2zxsl\") pod \"praxis-route-4429aa-kserve-668c498f79-4pb6w\" (UID: \"13089e4b-aad1-4aad-a869-1ef46d93bd6e\") " pod="ai-tenant-e2e-praxis-a-4429aa/praxis-route-4429aa-kserve-668c498f79-4pb6w" Nov 03 14:22:03.598046 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:03.597771 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13089e4b-aad1-4aad-a869-1ef46d93bd6e-tls-certs\") pod \"praxis-route-4429aa-kserve-668c498f79-4pb6w\" (UID: \"13089e4b-aad1-4aad-a869-1ef46d93bd6e\") " pod="ai-tenant-e2e-praxis-a-4429aa/praxis-route-4429aa-kserve-668c498f79-4pb6w" Nov 03 14:22:03.598046 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:03.597804 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13089e4b-aad1-4aad-a869-1ef46d93bd6e-home\") pod \"praxis-route-4429aa-kserve-668c498f79-4pb6w\" (UID: \"13089e4b-aad1-4aad-a869-1ef46d93bd6e\") " pod="ai-tenant-e2e-praxis-a-4429aa/praxis-route-4429aa-kserve-668c498f79-4pb6w" Nov 03 14:22:03.598046 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:03.597823 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13089e4b-aad1-4aad-a869-1ef46d93bd6e-model-cache\") pod \"praxis-route-4429aa-kserve-668c498f79-4pb6w\" (UID: \"13089e4b-aad1-4aad-a869-1ef46d93bd6e\") " pod="ai-tenant-e2e-praxis-a-4429aa/praxis-route-4429aa-kserve-668c498f79-4pb6w" Nov 03 14:22:03.699377 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:03.699331 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13089e4b-aad1-4aad-a869-1ef46d93bd6e-dshm\") pod \"praxis-route-4429aa-kserve-668c498f79-4pb6w\" (UID: \"13089e4b-aad1-4aad-a869-1ef46d93bd6e\") " pod="ai-tenant-e2e-praxis-a-4429aa/praxis-route-4429aa-kserve-668c498f79-4pb6w" Nov 03 14:22:03.699600 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:03.699409 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13089e4b-aad1-4aad-a869-1ef46d93bd6e-tmp-dir\") pod \"praxis-route-4429aa-kserve-668c498f79-4pb6w\" (UID: \"13089e4b-aad1-4aad-a869-1ef46d93bd6e\") " pod="ai-tenant-e2e-praxis-a-4429aa/praxis-route-4429aa-kserve-668c498f79-4pb6w" Nov 03 14:22:03.699600 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:03.699446 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zxsl\" (UniqueName: \"kubernetes.io/projected/13089e4b-aad1-4aad-a869-1ef46d93bd6e-kube-api-access-2zxsl\") pod \"praxis-route-4429aa-kserve-668c498f79-4pb6w\" (UID: \"13089e4b-aad1-4aad-a869-1ef46d93bd6e\") " pod="ai-tenant-e2e-praxis-a-4429aa/praxis-route-4429aa-kserve-668c498f79-4pb6w" Nov 03 14:22:03.699600 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:03.699536 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13089e4b-aad1-4aad-a869-1ef46d93bd6e-tls-certs\") pod \"praxis-route-4429aa-kserve-668c498f79-4pb6w\" (UID: \"13089e4b-aad1-4aad-a869-1ef46d93bd6e\") " pod="ai-tenant-e2e-praxis-a-4429aa/praxis-route-4429aa-kserve-668c498f79-4pb6w" Nov 03 14:22:03.699600 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:03.699564 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13089e4b-aad1-4aad-a869-1ef46d93bd6e-home\") pod \"praxis-route-4429aa-kserve-668c498f79-4pb6w\" (UID: \"13089e4b-aad1-4aad-a869-1ef46d93bd6e\") " pod="ai-tenant-e2e-praxis-a-4429aa/praxis-route-4429aa-kserve-668c498f79-4pb6w" Nov 03 14:22:03.699600 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:03.699590 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13089e4b-aad1-4aad-a869-1ef46d93bd6e-model-cache\") pod \"praxis-route-4429aa-kserve-668c498f79-4pb6w\" (UID: \"13089e4b-aad1-4aad-a869-1ef46d93bd6e\") " pod="ai-tenant-e2e-praxis-a-4429aa/praxis-route-4429aa-kserve-668c498f79-4pb6w" Nov 03 14:22:03.699921 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:03.699898 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13089e4b-aad1-4aad-a869-1ef46d93bd6e-tmp-dir\") pod \"praxis-route-4429aa-kserve-668c498f79-4pb6w\" (UID: \"13089e4b-aad1-4aad-a869-1ef46d93bd6e\") " pod="ai-tenant-e2e-praxis-a-4429aa/praxis-route-4429aa-kserve-668c498f79-4pb6w" Nov 03 14:22:03.700004 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:03.699949 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13089e4b-aad1-4aad-a869-1ef46d93bd6e-model-cache\") pod \"praxis-route-4429aa-kserve-668c498f79-4pb6w\" (UID: \"13089e4b-aad1-4aad-a869-1ef46d93bd6e\") " pod="ai-tenant-e2e-praxis-a-4429aa/praxis-route-4429aa-kserve-668c498f79-4pb6w" Nov 03 14:22:03.700161 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:03.700123 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13089e4b-aad1-4aad-a869-1ef46d93bd6e-home\") pod \"praxis-route-4429aa-kserve-668c498f79-4pb6w\" (UID: \"13089e4b-aad1-4aad-a869-1ef46d93bd6e\") " pod="ai-tenant-e2e-praxis-a-4429aa/praxis-route-4429aa-kserve-668c498f79-4pb6w" Nov 03 14:22:03.702298 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:03.702270 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13089e4b-aad1-4aad-a869-1ef46d93bd6e-dshm\") pod \"praxis-route-4429aa-kserve-668c498f79-4pb6w\" (UID: \"13089e4b-aad1-4aad-a869-1ef46d93bd6e\") " pod="ai-tenant-e2e-praxis-a-4429aa/praxis-route-4429aa-kserve-668c498f79-4pb6w" Nov 03 14:22:03.702565 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:03.702539 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13089e4b-aad1-4aad-a869-1ef46d93bd6e-tls-certs\") pod \"praxis-route-4429aa-kserve-668c498f79-4pb6w\" (UID: \"13089e4b-aad1-4aad-a869-1ef46d93bd6e\") " pod="ai-tenant-e2e-praxis-a-4429aa/praxis-route-4429aa-kserve-668c498f79-4pb6w" Nov 03 14:22:03.732823 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:03.732743 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zxsl\" (UniqueName: \"kubernetes.io/projected/13089e4b-aad1-4aad-a869-1ef46d93bd6e-kube-api-access-2zxsl\") pod \"praxis-route-4429aa-kserve-668c498f79-4pb6w\" (UID: \"13089e4b-aad1-4aad-a869-1ef46d93bd6e\") " pod="ai-tenant-e2e-praxis-a-4429aa/praxis-route-4429aa-kserve-668c498f79-4pb6w" Nov 03 14:22:03.792040 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:03.792008 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-4429aa/praxis-route-4429aa-kserve-668c498f79-4pb6w" Nov 03 14:22:04.099460 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:04.099420 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-4429aa/praxis-route-4429aa-kserve-668c498f79-4pb6w"] Nov 03 14:22:04.101907 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:22:04.101872 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13089e4b_aad1_4aad_a869_1ef46d93bd6e.slice/crio-c9492eeaf66c12c6590dff02252739ef67546e706101208951f054f6845f06f1 WatchSource:0}: Error finding container c9492eeaf66c12c6590dff02252739ef67546e706101208951f054f6845f06f1: Status 404 returned error can't find the container with id c9492eeaf66c12c6590dff02252739ef67546e706101208951f054f6845f06f1 Nov 03 14:22:04.104033 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:04.104011 5060 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 14:22:04.770510 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:04.770474 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-4429aa/praxis-route-4429aa-kserve-668c498f79-4pb6w" event={"ID":"13089e4b-aad1-4aad-a869-1ef46d93bd6e","Type":"ContainerStarted","Data":"c9492eeaf66c12c6590dff02252739ef67546e706101208951f054f6845f06f1"} Nov 03 14:22:06.787532 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:06.787491 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-4429aa/praxis-route-4429aa-kserve-668c498f79-4pb6w" event={"ID":"13089e4b-aad1-4aad-a869-1ef46d93bd6e","Type":"ContainerStarted","Data":"73874d214748f06bbb1041ae53874fefe47bf131c382d25fbcdd530d2c258f73"} Nov 03 14:22:06.792875 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:06.792846 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-4429aa/praxis-route-4429aa-kserve-668c498f79-4pb6w" Nov 03 14:22:06.849382 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:06.849333 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-4429aa/praxis-route-4429aa-kserve-668c498f79-4pb6w" podStartSLOduration=2.008846475 podStartE2EDuration="3.849319021s" totalImagesPullingTime="1.840472546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:22:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:22:06.848153743 +0000 UTC m=+660.736943325" watchObservedRunningTime="2026-11-03 14:22:06.849319021 +0000 UTC m=+660.738108601" Nov 03 14:22:09.450180 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:22:09.450143 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:22:13.793022 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:13.792983 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-4429aa/praxis-route-4429aa-kserve-668c498f79-4pb6w" Nov 03 14:22:13.805513 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:13.805486 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-4429aa/praxis-route-4429aa-kserve-668c498f79-4pb6w" Nov 03 14:22:13.849962 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:13.848324 5060 generic.go:356] "Generic (PLEG): container finished" podID="6c8d7e93-3520-4a18-916a-ac77ae09f885" containerID="572d43f258e5370b50d58feb6d259d6c0b88b376677bad0e8cc855d1e9381b29" exitCode=0 Nov 03 14:22:13.849962 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:13.848451 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" event={"ID":"6c8d7e93-3520-4a18-916a-ac77ae09f885","Type":"ContainerDied","Data":"572d43f258e5370b50d58feb6d259d6c0b88b376677bad0e8cc855d1e9381b29"} Nov 03 14:22:13.849962 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:13.849593 5060 scope.go:122] "RemoveContainer" containerID="572d43f258e5370b50d58feb6d259d6c0b88b376677bad0e8cc855d1e9381b29" Nov 03 14:22:13.851001 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:13.850969 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:22:13.851129 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:13.851049 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:22:13.851129 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:13.851097 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:22:13.861602 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:13.861571 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-4429aa/praxis-route-4429aa-kserve-668c498f79-4pb6w" Nov 03 14:22:14.855768 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:14.855734 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" event={"ID":"6c8d7e93-3520-4a18-916a-ac77ae09f885","Type":"ContainerStarted","Data":"3feae28bb8fa9d9deb1e32ba02f68f4da3523e3787f19f09502d8ab81856ba5b"} Nov 03 14:22:15.692973 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:15.692936 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" Nov 03 14:22:15.694147 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:15.694126 5060 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Nov 03 14:22:15.694205 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:15.694171 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" podUID="6c8d7e93-3520-4a18-916a-ac77ae09f885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Nov 03 14:22:16.693090 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:16.693057 5060 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Nov 03 14:22:16.693446 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:16.693106 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" podUID="6c8d7e93-3520-4a18-916a-ac77ae09f885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Nov 03 14:22:17.625912 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:17.624927 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" Nov 03 14:22:17.692920 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:17.692887 5060 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Nov 03 14:22:17.693082 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:17.692938 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" podUID="6c8d7e93-3520-4a18-916a-ac77ae09f885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Nov 03 14:22:18.692992 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:18.692956 5060 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Nov 03 14:22:18.693360 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:18.693024 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" podUID="6c8d7e93-3520-4a18-916a-ac77ae09f885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Nov 03 14:22:19.692499 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:19.692465 5060 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Nov 03 14:22:19.692686 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:19.692536 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" podUID="6c8d7e93-3520-4a18-916a-ac77ae09f885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Nov 03 14:22:20.696605 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:20.696570 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" Nov 03 14:22:20.697503 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:20.697484 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" Nov 03 14:22:37.633373 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:37.633337 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-156ae1/e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr"] Nov 03 14:22:37.638161 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:37.638129 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr" Nov 03 14:22:37.667448 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:37.667410 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-156ae1\"/\"kube-root-ca.crt\"" Nov 03 14:22:37.667604 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:37.667475 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-156ae1\"/\"default-dockercfg-wqn4r\"" Nov 03 14:22:37.667604 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:37.667543 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-156ae1\"/\"openshift-service-ca.crt\"" Nov 03 14:22:37.668157 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:37.668142 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-156ae1\"/\"e2e-conflict-b-1c60a22c-kserve-self-signed-certs\"" Nov 03 14:22:37.671100 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:37.671077 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-156ae1/e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr"] Nov 03 14:22:37.756724 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:37.756682 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4-tmp-dir\") pod \"e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr\" (UID: \"94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4\") " pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr" Nov 03 14:22:37.756960 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:37.756817 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4-dshm\") pod \"e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr\" (UID: \"94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4\") " pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr" Nov 03 14:22:37.756960 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:37.756944 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4-model-cache\") pod \"e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr\" (UID: \"94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4\") " pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr" Nov 03 14:22:37.757067 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:37.757018 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4-tls-certs\") pod \"e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr\" (UID: \"94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4\") " pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr" Nov 03 14:22:37.757111 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:37.757094 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn5b2\" (UniqueName: \"kubernetes.io/projected/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4-kube-api-access-hn5b2\") pod \"e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr\" (UID: \"94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4\") " pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr" Nov 03 14:22:37.757202 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:37.757186 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4-home\") pod \"e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr\" (UID: \"94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4\") " pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr" Nov 03 14:22:37.858679 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:37.858633 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn5b2\" (UniqueName: \"kubernetes.io/projected/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4-kube-api-access-hn5b2\") pod \"e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr\" (UID: \"94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4\") " pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr" Nov 03 14:22:37.858679 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:37.858694 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4-home\") pod \"e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr\" (UID: \"94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4\") " pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr" Nov 03 14:22:37.858977 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:37.858740 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4-tmp-dir\") pod \"e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr\" (UID: \"94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4\") " pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr" Nov 03 14:22:37.858977 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:37.858768 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4-dshm\") pod \"e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr\" (UID: \"94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4\") " pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr" Nov 03 14:22:37.858977 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:37.858819 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4-model-cache\") pod \"e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr\" (UID: \"94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4\") " pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr" Nov 03 14:22:37.858977 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:37.858877 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4-tls-certs\") pod \"e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr\" (UID: \"94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4\") " pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr" Nov 03 14:22:37.859294 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:37.859267 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4-tmp-dir\") pod \"e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr\" (UID: \"94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4\") " pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr" Nov 03 14:22:37.859390 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:37.859365 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4-model-cache\") pod \"e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr\" (UID: \"94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4\") " pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr" Nov 03 14:22:37.859599 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:37.859568 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4-home\") pod \"e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr\" (UID: \"94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4\") " pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr" Nov 03 14:22:37.861351 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:37.861330 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4-dshm\") pod \"e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr\" (UID: \"94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4\") " pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr" Nov 03 14:22:37.862998 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:37.862977 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4-tls-certs\") pod \"e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr\" (UID: \"94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4\") " pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr" Nov 03 14:22:37.881170 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:37.881142 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn5b2\" (UniqueName: \"kubernetes.io/projected/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4-kube-api-access-hn5b2\") pod \"e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr\" (UID: \"94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4\") " pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr" Nov 03 14:22:37.953441 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:37.953326 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr" Nov 03 14:22:38.255438 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:38.255416 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-156ae1/e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr"] Nov 03 14:22:38.257817 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:22:38.257791 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94ccad46_ebe5_4a0b_9cff_6efaf6ba66b4.slice/crio-9c6662d926cdd5d5ccf698a30ea7ee5df7ee3866e8a30b6185c2eaf86aad179d WatchSource:0}: Error finding container 9c6662d926cdd5d5ccf698a30ea7ee5df7ee3866e8a30b6185c2eaf86aad179d: Status 404 returned error can't find the container with id 9c6662d926cdd5d5ccf698a30ea7ee5df7ee3866e8a30b6185c2eaf86aad179d Nov 03 14:22:38.987707 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:38.987670 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr" event={"ID":"94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4","Type":"ContainerStarted","Data":"90544d737dbfc552c79a3358ab4618fa11277154074aa43b1170ef2919d1175f"} Nov 03 14:22:38.988159 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:38.987717 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr" event={"ID":"94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4","Type":"ContainerStarted","Data":"9c6662d926cdd5d5ccf698a30ea7ee5df7ee3866e8a30b6185c2eaf86aad179d"} Nov 03 14:22:39.032802 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:39.032733 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr" podStartSLOduration=2.032712587 podStartE2EDuration="2.032712587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:22:39.030814467 +0000 UTC m=+692.919604049" watchObservedRunningTime="2026-11-03 14:22:39.032712587 +0000 UTC m=+692.921502176" Nov 03 14:22:39.438794 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:22:39.438742 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:22:39.949350 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:39.949313 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc"] Nov 03 14:22:39.953333 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:39.953308 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" Nov 03 14:22:39.953988 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:39.953960 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr" Nov 03 14:22:39.974476 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:39.974456 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-86c2e2-openshift-default-dockercfg-7dv7n\"" Nov 03 14:22:39.989623 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:39.989587 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc"] Nov 03 14:22:40.081258 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:40.081221 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b671512d-59b5-4ddd-959d-1d9150d41268-istio-envoy\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc\" (UID: \"b671512d-59b5-4ddd-959d-1d9150d41268\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" Nov 03 14:22:40.081258 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:40.081257 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b671512d-59b5-4ddd-959d-1d9150d41268-credential-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc\" (UID: \"b671512d-59b5-4ddd-959d-1d9150d41268\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" Nov 03 14:22:40.081502 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:40.081368 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b671512d-59b5-4ddd-959d-1d9150d41268-workload-certs\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc\" (UID: \"b671512d-59b5-4ddd-959d-1d9150d41268\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" Nov 03 14:22:40.081502 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:40.081469 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzlvn\" (UniqueName: \"kubernetes.io/projected/b671512d-59b5-4ddd-959d-1d9150d41268-kube-api-access-vzlvn\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc\" (UID: \"b671512d-59b5-4ddd-959d-1d9150d41268\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" Nov 03 14:22:40.081616 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:40.081513 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b671512d-59b5-4ddd-959d-1d9150d41268-istio-podinfo\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc\" (UID: \"b671512d-59b5-4ddd-959d-1d9150d41268\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" Nov 03 14:22:40.081699 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:40.081675 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b671512d-59b5-4ddd-959d-1d9150d41268-istio-token\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc\" (UID: \"b671512d-59b5-4ddd-959d-1d9150d41268\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" Nov 03 14:22:40.081826 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:40.081773 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b671512d-59b5-4ddd-959d-1d9150d41268-istiod-ca-cert\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc\" (UID: \"b671512d-59b5-4ddd-959d-1d9150d41268\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" Nov 03 14:22:40.081826 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:40.081816 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b671512d-59b5-4ddd-959d-1d9150d41268-workload-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc\" (UID: \"b671512d-59b5-4ddd-959d-1d9150d41268\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" Nov 03 14:22:40.081949 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:40.081915 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b671512d-59b5-4ddd-959d-1d9150d41268-istio-data\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc\" (UID: \"b671512d-59b5-4ddd-959d-1d9150d41268\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" Nov 03 14:22:40.182813 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:40.182775 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b671512d-59b5-4ddd-959d-1d9150d41268-istiod-ca-cert\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc\" (UID: \"b671512d-59b5-4ddd-959d-1d9150d41268\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" Nov 03 14:22:40.182813 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:40.182814 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b671512d-59b5-4ddd-959d-1d9150d41268-workload-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc\" (UID: \"b671512d-59b5-4ddd-959d-1d9150d41268\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" Nov 03 14:22:40.183075 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:40.182868 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b671512d-59b5-4ddd-959d-1d9150d41268-istio-data\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc\" (UID: \"b671512d-59b5-4ddd-959d-1d9150d41268\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" Nov 03 14:22:40.183075 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:40.182900 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b671512d-59b5-4ddd-959d-1d9150d41268-istio-envoy\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc\" (UID: \"b671512d-59b5-4ddd-959d-1d9150d41268\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" Nov 03 14:22:40.183075 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:40.182926 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b671512d-59b5-4ddd-959d-1d9150d41268-credential-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc\" (UID: \"b671512d-59b5-4ddd-959d-1d9150d41268\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" Nov 03 14:22:40.183075 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:40.182958 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b671512d-59b5-4ddd-959d-1d9150d41268-workload-certs\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc\" (UID: \"b671512d-59b5-4ddd-959d-1d9150d41268\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" Nov 03 14:22:40.183075 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:40.183009 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzlvn\" (UniqueName: \"kubernetes.io/projected/b671512d-59b5-4ddd-959d-1d9150d41268-kube-api-access-vzlvn\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc\" (UID: \"b671512d-59b5-4ddd-959d-1d9150d41268\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" Nov 03 14:22:40.183075 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:40.183039 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b671512d-59b5-4ddd-959d-1d9150d41268-istio-podinfo\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc\" (UID: \"b671512d-59b5-4ddd-959d-1d9150d41268\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" Nov 03 14:22:40.183441 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:40.183105 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b671512d-59b5-4ddd-959d-1d9150d41268-istio-token\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc\" (UID: \"b671512d-59b5-4ddd-959d-1d9150d41268\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" Nov 03 14:22:40.183441 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:40.183296 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b671512d-59b5-4ddd-959d-1d9150d41268-credential-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc\" (UID: \"b671512d-59b5-4ddd-959d-1d9150d41268\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" Nov 03 14:22:40.183441 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:40.183362 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b671512d-59b5-4ddd-959d-1d9150d41268-workload-certs\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc\" (UID: \"b671512d-59b5-4ddd-959d-1d9150d41268\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" Nov 03 14:22:40.183648 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:40.183625 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b671512d-59b5-4ddd-959d-1d9150d41268-istiod-ca-cert\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc\" (UID: \"b671512d-59b5-4ddd-959d-1d9150d41268\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" Nov 03 14:22:40.183812 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:40.183790 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b671512d-59b5-4ddd-959d-1d9150d41268-workload-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc\" (UID: \"b671512d-59b5-4ddd-959d-1d9150d41268\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" Nov 03 14:22:40.183909 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:40.183871 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b671512d-59b5-4ddd-959d-1d9150d41268-istio-data\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc\" (UID: \"b671512d-59b5-4ddd-959d-1d9150d41268\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" Nov 03 14:22:40.185741 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:40.185713 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b671512d-59b5-4ddd-959d-1d9150d41268-istio-envoy\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc\" (UID: \"b671512d-59b5-4ddd-959d-1d9150d41268\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" Nov 03 14:22:40.185989 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:40.185971 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b671512d-59b5-4ddd-959d-1d9150d41268-istio-podinfo\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc\" (UID: \"b671512d-59b5-4ddd-959d-1d9150d41268\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" Nov 03 14:22:40.207531 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:40.207463 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b671512d-59b5-4ddd-959d-1d9150d41268-istio-token\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc\" (UID: \"b671512d-59b5-4ddd-959d-1d9150d41268\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" Nov 03 14:22:40.207635 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:40.207609 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzlvn\" (UniqueName: \"kubernetes.io/projected/b671512d-59b5-4ddd-959d-1d9150d41268-kube-api-access-vzlvn\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc\" (UID: \"b671512d-59b5-4ddd-959d-1d9150d41268\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" Nov 03 14:22:40.269998 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:40.269957 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" Nov 03 14:22:40.590329 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:40.590301 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc"] Nov 03 14:22:40.592375 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:22:40.592334 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb671512d_59b5_4ddd_959d_1d9150d41268.slice/crio-1d31b71826d40bf0050ff127459e3b16e7f9958631a5727e0c88fbfbbf80fd20 WatchSource:0}: Error finding container 1d31b71826d40bf0050ff127459e3b16e7f9958631a5727e0c88fbfbbf80fd20: Status 404 returned error can't find the container with id 1d31b71826d40bf0050ff127459e3b16e7f9958631a5727e0c88fbfbbf80fd20 Nov 03 14:22:40.594685 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:40.594653 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:22:40.594759 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:40.594716 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:22:40.594759 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:40.594746 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:22:41.002423 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:41.002385 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" event={"ID":"b671512d-59b5-4ddd-959d-1d9150d41268","Type":"ContainerStarted","Data":"5e9e8520dd24462be6b1e66bcad82e3f8d92ffebe4a2569073dd63c91f586a01"} Nov 03 14:22:41.002869 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:41.002455 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" event={"ID":"b671512d-59b5-4ddd-959d-1d9150d41268","Type":"ContainerStarted","Data":"1d31b71826d40bf0050ff127459e3b16e7f9958631a5727e0c88fbfbbf80fd20"} Nov 03 14:22:41.078604 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:41.078556 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" podStartSLOduration=2.078541408 podStartE2EDuration="2.078541408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:22:41.064143158 +0000 UTC m=+694.952932740" watchObservedRunningTime="2026-11-03 14:22:41.078541408 +0000 UTC m=+694.967330989" Nov 03 14:22:41.270616 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:41.270536 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" Nov 03 14:22:41.272097 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:41.272070 5060 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 03 14:22:41.272242 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:41.272115 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" podUID="b671512d-59b5-4ddd-959d-1d9150d41268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 03 14:22:42.271113 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:42.271073 5060 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 03 14:22:42.271506 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:42.271150 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" podUID="b671512d-59b5-4ddd-959d-1d9150d41268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 03 14:22:43.270403 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:43.270360 5060 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 03 14:22:43.270582 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:43.270434 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" podUID="b671512d-59b5-4ddd-959d-1d9150d41268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 03 14:22:44.271189 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:44.271154 5060 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 03 14:22:44.271811 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:44.271212 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" podUID="b671512d-59b5-4ddd-959d-1d9150d41268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 03 14:22:45.270362 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:45.270327 5060 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 03 14:22:45.270564 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:45.270379 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" podUID="b671512d-59b5-4ddd-959d-1d9150d41268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 03 14:22:46.270797 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:46.270764 5060 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 03 14:22:46.271202 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:46.270811 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" podUID="b671512d-59b5-4ddd-959d-1d9150d41268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 03 14:22:47.270421 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:47.270383 5060 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 03 14:22:47.270606 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:47.270436 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" podUID="b671512d-59b5-4ddd-959d-1d9150d41268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 03 14:22:47.953645 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:47.953600 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr" Nov 03 14:22:47.966103 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:47.966075 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr" Nov 03 14:22:48.082041 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:48.082004 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr" Nov 03 14:22:48.274968 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:48.274939 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" Nov 03 14:22:48.275372 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:48.275348 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" Nov 03 14:22:48.276107 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:48.276089 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" Nov 03 14:22:53.120767 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:53.120731 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91aabadb-gwns8"] Nov 03 14:22:53.124384 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:53.124361 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91aabadb-gwns8" Nov 03 14:22:53.147453 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:53.147428 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91aabadb-gwns8"] Nov 03 14:22:53.157074 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:53.157047 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:22:53.157195 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:53.157053 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:22:53.157195 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:53.157091 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\"" Nov 03 14:22:53.157195 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:53.157091 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:22:53.238472 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:53.238440 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sphg9\" (UniqueName: \"kubernetes.io/projected/8f29f3d1-2c91-4403-ac7c-f404fb688ac4-kube-api-access-sphg9\") pod \"maas-api-revoke-sub-91aabadb-gwns8\" (UID: \"8f29f3d1-2c91-4403-ac7c-f404fb688ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91aabadb-gwns8" Nov 03 14:22:53.238667 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:53.238568 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f29f3d1-2c91-4403-ac7c-f404fb688ac4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91aabadb-gwns8\" (UID: \"8f29f3d1-2c91-4403-ac7c-f404fb688ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91aabadb-gwns8" Nov 03 14:22:53.238720 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:53.238678 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f29f3d1-2c91-4403-ac7c-f404fb688ac4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91aabadb-gwns8\" (UID: \"8f29f3d1-2c91-4403-ac7c-f404fb688ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91aabadb-gwns8" Nov 03 14:22:53.340400 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:53.340360 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sphg9\" (UniqueName: \"kubernetes.io/projected/8f29f3d1-2c91-4403-ac7c-f404fb688ac4-kube-api-access-sphg9\") pod \"maas-api-revoke-sub-91aabadb-gwns8\" (UID: \"8f29f3d1-2c91-4403-ac7c-f404fb688ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91aabadb-gwns8" Nov 03 14:22:53.340589 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:53.340474 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f29f3d1-2c91-4403-ac7c-f404fb688ac4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91aabadb-gwns8\" (UID: \"8f29f3d1-2c91-4403-ac7c-f404fb688ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91aabadb-gwns8" Nov 03 14:22:53.340589 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:53.340540 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f29f3d1-2c91-4403-ac7c-f404fb688ac4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91aabadb-gwns8\" (UID: \"8f29f3d1-2c91-4403-ac7c-f404fb688ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91aabadb-gwns8" Nov 03 14:22:53.341503 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:53.341472 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f29f3d1-2c91-4403-ac7c-f404fb688ac4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91aabadb-gwns8\" (UID: \"8f29f3d1-2c91-4403-ac7c-f404fb688ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91aabadb-gwns8" Nov 03 14:22:53.343168 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:53.343139 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f29f3d1-2c91-4403-ac7c-f404fb688ac4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91aabadb-gwns8\" (UID: \"8f29f3d1-2c91-4403-ac7c-f404fb688ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91aabadb-gwns8" Nov 03 14:22:53.367458 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:53.367428 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sphg9\" (UniqueName: \"kubernetes.io/projected/8f29f3d1-2c91-4403-ac7c-f404fb688ac4-kube-api-access-sphg9\") pod \"maas-api-revoke-sub-91aabadb-gwns8\" (UID: \"8f29f3d1-2c91-4403-ac7c-f404fb688ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91aabadb-gwns8" Nov 03 14:22:53.438543 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:53.438451 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91aabadb-gwns8" Nov 03 14:22:53.775999 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:53.775974 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91aabadb-gwns8"] Nov 03 14:22:53.778152 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:22:53.778113 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f29f3d1_2c91_4403_ac7c_f404fb688ac4.slice/crio-b3b18ddb01ba1dcb7b8b8bdd6e6eeec0bfe7bdf52bc59e19463bea769aa6911e WatchSource:0}: Error finding container b3b18ddb01ba1dcb7b8b8bdd6e6eeec0bfe7bdf52bc59e19463bea769aa6911e: Status 404 returned error can't find the container with id b3b18ddb01ba1dcb7b8b8bdd6e6eeec0bfe7bdf52bc59e19463bea769aa6911e Nov 03 14:22:54.115753 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:54.115612 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91aabadb-gwns8" event={"ID":"8f29f3d1-2c91-4403-ac7c-f404fb688ac4","Type":"ContainerStarted","Data":"b3b18ddb01ba1dcb7b8b8bdd6e6eeec0bfe7bdf52bc59e19463bea769aa6911e"} Nov 03 14:22:57.137786 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:57.137745 5060 generic.go:356] "Generic (PLEG): container finished" podID="8f29f3d1-2c91-4403-ac7c-f404fb688ac4" containerID="397f93e82ffb30e8f057a650c243eca5bdad33f39b2cb64a9406281149c8da15" exitCode=0 Nov 03 14:22:57.138302 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:57.137795 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91aabadb-gwns8" event={"ID":"8f29f3d1-2c91-4403-ac7c-f404fb688ac4","Type":"ContainerDied","Data":"397f93e82ffb30e8f057a650c243eca5bdad33f39b2cb64a9406281149c8da15"} Nov 03 14:22:57.962867 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:57.962815 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-4429aa/praxis-route-4429aa-kserve-668c498f79-4pb6w"] Nov 03 14:22:58.285445 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:58.285421 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91aabadb-gwns8" Nov 03 14:22:58.409798 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:58.409757 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f29f3d1-2c91-4403-ac7c-f404fb688ac4-kube-api-access-sphg9\" (UniqueName: \"kubernetes.io/projected/8f29f3d1-2c91-4403-ac7c-f404fb688ac4-kube-api-access-sphg9\") pod \"8f29f3d1-2c91-4403-ac7c-f404fb688ac4\" (UID: \"8f29f3d1-2c91-4403-ac7c-f404fb688ac4\") " Nov 03 14:22:58.410023 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:58.409809 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f29f3d1-2c91-4403-ac7c-f404fb688ac4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f29f3d1-2c91-4403-ac7c-f404fb688ac4-maas-api-xks-ca\") pod \"8f29f3d1-2c91-4403-ac7c-f404fb688ac4\" (UID: \"8f29f3d1-2c91-4403-ac7c-f404fb688ac4\") " Nov 03 14:22:58.410023 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:58.409972 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f29f3d1-2c91-4403-ac7c-f404fb688ac4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f29f3d1-2c91-4403-ac7c-f404fb688ac4-maas-api-openshift-ca\") pod \"8f29f3d1-2c91-4403-ac7c-f404fb688ac4\" (UID: \"8f29f3d1-2c91-4403-ac7c-f404fb688ac4\") " Nov 03 14:22:58.410323 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:58.410291 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f29f3d1-2c91-4403-ac7c-f404fb688ac4-maas-api-openshift-ca" pod "8f29f3d1-2c91-4403-ac7c-f404fb688ac4" (UID: "8f29f3d1-2c91-4403-ac7c-f404fb688ac4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:22:58.412192 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:58.412161 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f29f3d1-2c91-4403-ac7c-f404fb688ac4-maas-api-xks-ca" pod "8f29f3d1-2c91-4403-ac7c-f404fb688ac4" (UID: "8f29f3d1-2c91-4403-ac7c-f404fb688ac4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:22:58.412349 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:58.412324 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f29f3d1-2c91-4403-ac7c-f404fb688ac4-kube-api-access-sphg9" pod "8f29f3d1-2c91-4403-ac7c-f404fb688ac4" (UID: "8f29f3d1-2c91-4403-ac7c-f404fb688ac4"). InnerVolumeSpecName "kube-api-access-sphg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:22:58.511604 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:58.511570 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f29f3d1-2c91-4403-ac7c-f404fb688ac4-maas-api-openshift-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:22:58.511604 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:58.511600 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sphg9\" (UniqueName: \"kubernetes.io/projected/8f29f3d1-2c91-4403-ac7c-f404fb688ac4-kube-api-access-sphg9\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:22:58.511604 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:58.511612 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f29f3d1-2c91-4403-ac7c-f404fb688ac4-maas-api-xks-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:22:59.151397 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:59.151369 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91aabadb-gwns8" Nov 03 14:22:59.151566 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:59.151399 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91aabadb-gwns8" event={"ID":"8f29f3d1-2c91-4403-ac7c-f404fb688ac4","Type":"ContainerDied","Data":"b3b18ddb01ba1dcb7b8b8bdd6e6eeec0bfe7bdf52bc59e19463bea769aa6911e"} Nov 03 14:22:59.151566 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:22:59.151451 5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3b18ddb01ba1dcb7b8b8bdd6e6eeec0bfe7bdf52bc59e19463bea769aa6911e" Nov 03 14:23:02.437684 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:02.437637 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-156ae1/e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr"] Nov 03 14:23:08.024438 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:08.024404 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7316-4069cb"] Nov 03 14:23:08.024887 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:08.024873 5060 cpu_manager.go:414] "containerMap: removing container" podUID="8f29f3d1-2c91-4403-ac7c-f404fb688ac4" containerName="revoke-keys" Nov 03 14:23:08.024944 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:08.024888 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f29f3d1-2c91-4403-ac7c-f404fb688ac4" containerName="revoke-keys" Nov 03 14:23:08.024985 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:08.024975 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f29f3d1-2c91-4403-ac7c-f404fb688ac4" containerName="revoke-keys" Nov 03 14:23:08.029304 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:08.029286 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7316-4069cb" Nov 03 14:23:08.050715 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:08.050681 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7316-4069cb"] Nov 03 14:23:08.060586 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:08.060563 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-9sm86\"" Nov 03 14:23:08.112905 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:08.112866 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztgvh\" (UniqueName: \"kubernetes.io/projected/71d5e661-5675-4b1c-b1e5-24a3c65829e0-kube-api-access-ztgvh\") pod \"test-curl-7316-4069cb\" (UID: \"71d5e661-5675-4b1c-b1e5-24a3c65829e0\") " pod="opendatahub/test-curl-7316-4069cb" Nov 03 14:23:08.214122 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:08.214090 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztgvh\" (UniqueName: \"kubernetes.io/projected/71d5e661-5675-4b1c-b1e5-24a3c65829e0-kube-api-access-ztgvh\") pod \"test-curl-7316-4069cb\" (UID: \"71d5e661-5675-4b1c-b1e5-24a3c65829e0\") " pod="opendatahub/test-curl-7316-4069cb" Nov 03 14:23:08.235852 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:08.235814 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztgvh\" (UniqueName: \"kubernetes.io/projected/71d5e661-5675-4b1c-b1e5-24a3c65829e0-kube-api-access-ztgvh\") pod \"test-curl-7316-4069cb\" (UID: \"71d5e661-5675-4b1c-b1e5-24a3c65829e0\") " pod="opendatahub/test-curl-7316-4069cb" Nov 03 14:23:08.341541 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:08.341445 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7316-4069cb" Nov 03 14:23:08.645762 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:08.645731 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7316-4069cb"] Nov 03 14:23:08.648430 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:23:08.648386 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71d5e661_5675_4b1c_b1e5_24a3c65829e0.slice/crio-cc652da59ea6c230c7ad309d27c7da88290f6ce78f28120602c3f738393f2849 WatchSource:0}: Error finding container cc652da59ea6c230c7ad309d27c7da88290f6ce78f28120602c3f738393f2849: Status 404 returned error can't find the container with id cc652da59ea6c230c7ad309d27c7da88290f6ce78f28120602c3f738393f2849 Nov 03 14:23:09.210105 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:09.210068 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7316-4069cb" event={"ID":"71d5e661-5675-4b1c-b1e5-24a3c65829e0","Type":"ContainerStarted","Data":"cc652da59ea6c230c7ad309d27c7da88290f6ce78f28120602c3f738393f2849"} Nov 03 14:23:09.439494 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:23:09.439450 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:23:11.226584 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:11.226540 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7316-4069cb" event={"ID":"71d5e661-5675-4b1c-b1e5-24a3c65829e0","Type":"ContainerStarted","Data":"7709861185af960338ad14449398d400972ba748a237d9eefdf29370670ade14"} Nov 03 14:23:11.270034 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:11.269982 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7316-4069cb" podStartSLOduration=0.898769057 podStartE2EDuration="3.269967689s" totalImagesPullingTime="2.371198632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:23:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:23:11.268364321 +0000 UTC m=+725.157153902" watchObservedRunningTime="2026-11-03 14:23:11.269967689 +0000 UTC m=+725.158757269" Nov 03 14:23:12.129507 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:12.129470 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7316-4069cb"] Nov 03 14:23:12.135772 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:12.135729 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7316-4069cb"] Nov 03 14:23:12.287644 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:12.287610 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7316-83dcde"] Nov 03 14:23:12.288050 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:12.288039 5060 cpu_manager.go:414] "containerMap: removing container" podUID="71d5e661-5675-4b1c-b1e5-24a3c65829e0" containerName="test-curl-7316-4069cb" Nov 03 14:23:12.288088 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:12.288050 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71d5e661-5675-4b1c-b1e5-24a3c65829e0" containerName="test-curl-7316-4069cb" Nov 03 14:23:12.288156 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:12.288145 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="71d5e661-5675-4b1c-b1e5-24a3c65829e0" containerName="test-curl-7316-4069cb" Nov 03 14:23:12.291304 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:12.291288 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7316-83dcde" Nov 03 14:23:12.313997 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:12.313967 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7316-83dcde"] Nov 03 14:23:12.463261 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:12.463174 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gczcm\" (UniqueName: \"kubernetes.io/projected/8b440e41-f5ad-458d-a874-29d3a84eccf1-kube-api-access-gczcm\") pod \"test-curl-7316-83dcde\" (UID: \"8b440e41-f5ad-458d-a874-29d3a84eccf1\") " pod="opendatahub/test-curl-7316-83dcde" Nov 03 14:23:12.564068 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:12.564031 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gczcm\" (UniqueName: \"kubernetes.io/projected/8b440e41-f5ad-458d-a874-29d3a84eccf1-kube-api-access-gczcm\") pod \"test-curl-7316-83dcde\" (UID: \"8b440e41-f5ad-458d-a874-29d3a84eccf1\") " pod="opendatahub/test-curl-7316-83dcde" Nov 03 14:23:12.588586 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:12.588556 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gczcm\" (UniqueName: \"kubernetes.io/projected/8b440e41-f5ad-458d-a874-29d3a84eccf1-kube-api-access-gczcm\") pod \"test-curl-7316-83dcde\" (UID: \"8b440e41-f5ad-458d-a874-29d3a84eccf1\") " pod="opendatahub/test-curl-7316-83dcde" Nov 03 14:23:12.603507 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:12.603478 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7316-83dcde" Nov 03 14:23:12.913379 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:12.913278 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7316-83dcde"] Nov 03 14:23:12.916766 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:23:12.916731 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b440e41_f5ad_458d_a874_29d3a84eccf1.slice/crio-de05d5076693dab36e06a76898f1a568a7f58d1703fe8cda0c269cdd5fcfbd98 WatchSource:0}: Error finding container de05d5076693dab36e06a76898f1a568a7f58d1703fe8cda0c269cdd5fcfbd98: Status 404 returned error can't find the container with id de05d5076693dab36e06a76898f1a568a7f58d1703fe8cda0c269cdd5fcfbd98 Nov 03 14:23:12.983907 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:12.983822 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-4429aa/praxis-route-4429aa-kserve-668c498f79-4pb6w" podUID="13089e4b-aad1-4aad-a869-1ef46d93bd6e" containerName="main" containerID="cri-o://73874d214748f06bbb1041ae53874fefe47bf131c382d25fbcdd530d2c258f73" gracePeriod=45 Nov 03 14:23:13.243650 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:13.243424 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7316-83dcde" event={"ID":"8b440e41-f5ad-458d-a874-29d3a84eccf1","Type":"ContainerStarted","Data":"9e3f532e5cf4f253e3fcdbe48c3346439b6a0e514c012232996c65c773332766"} Nov 03 14:23:13.243650 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:13.243480 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7316-83dcde" event={"ID":"8b440e41-f5ad-458d-a874-29d3a84eccf1","Type":"ContainerStarted","Data":"de05d5076693dab36e06a76898f1a568a7f58d1703fe8cda0c269cdd5fcfbd98"} Nov 03 14:23:13.245541 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:13.245512 5060 generic.go:356] "Generic (PLEG): container finished" podID="13089e4b-aad1-4aad-a869-1ef46d93bd6e" containerID="73874d214748f06bbb1041ae53874fefe47bf131c382d25fbcdd530d2c258f73" exitCode=0 Nov 03 14:23:13.245683 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:13.245610 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-4429aa/praxis-route-4429aa-kserve-668c498f79-4pb6w" event={"ID":"13089e4b-aad1-4aad-a869-1ef46d93bd6e","Type":"ContainerDied","Data":"73874d214748f06bbb1041ae53874fefe47bf131c382d25fbcdd530d2c258f73"} Nov 03 14:23:13.245683 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:13.245662 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7316-4069cb" podUID="71d5e661-5675-4b1c-b1e5-24a3c65829e0" containerName="test-curl-7316-4069cb" containerID="cri-o://7709861185af960338ad14449398d400972ba748a237d9eefdf29370670ade14" gracePeriod=2 Nov 03 14:23:13.252742 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:13.252724 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-4429aa/praxis-route-4429aa-kserve-668c498f79-4pb6w" Nov 03 14:23:13.286074 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:13.286029 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7316-83dcde" podStartSLOduration=1.28601566 podStartE2EDuration="1.28601566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:23:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:23:13.283554736 +0000 UTC m=+727.172344317" watchObservedRunningTime="2026-11-03 14:23:13.28601566 +0000 UTC m=+727.174805241" Nov 03 14:23:13.371195 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:13.371161 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13089e4b-aad1-4aad-a869-1ef46d93bd6e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13089e4b-aad1-4aad-a869-1ef46d93bd6e-tmp-dir\") pod \"13089e4b-aad1-4aad-a869-1ef46d93bd6e\" (UID: \"13089e4b-aad1-4aad-a869-1ef46d93bd6e\") " Nov 03 14:23:13.371661 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:13.371230 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13089e4b-aad1-4aad-a869-1ef46d93bd6e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/13089e4b-aad1-4aad-a869-1ef46d93bd6e-dshm\") pod \"13089e4b-aad1-4aad-a869-1ef46d93bd6e\" (UID: \"13089e4b-aad1-4aad-a869-1ef46d93bd6e\") " Nov 03 14:23:13.371661 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:13.371261 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13089e4b-aad1-4aad-a869-1ef46d93bd6e-tls-certs\" (UniqueName: \"kubernetes.io/secret/13089e4b-aad1-4aad-a869-1ef46d93bd6e-tls-certs\") pod \"13089e4b-aad1-4aad-a869-1ef46d93bd6e\" (UID: \"13089e4b-aad1-4aad-a869-1ef46d93bd6e\") " Nov 03 14:23:13.371661 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:13.371311 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13089e4b-aad1-4aad-a869-1ef46d93bd6e-kube-api-access-2zxsl\" (UniqueName: \"kubernetes.io/projected/13089e4b-aad1-4aad-a869-1ef46d93bd6e-kube-api-access-2zxsl\") pod \"13089e4b-aad1-4aad-a869-1ef46d93bd6e\" (UID: \"13089e4b-aad1-4aad-a869-1ef46d93bd6e\") " Nov 03 14:23:13.371661 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:13.371338 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13089e4b-aad1-4aad-a869-1ef46d93bd6e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13089e4b-aad1-4aad-a869-1ef46d93bd6e-model-cache\") pod \"13089e4b-aad1-4aad-a869-1ef46d93bd6e\" (UID: \"13089e4b-aad1-4aad-a869-1ef46d93bd6e\") " Nov 03 14:23:13.371661 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:13.371455 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13089e4b-aad1-4aad-a869-1ef46d93bd6e-home\" (UniqueName: \"kubernetes.io/empty-dir/13089e4b-aad1-4aad-a869-1ef46d93bd6e-home\") pod \"13089e4b-aad1-4aad-a869-1ef46d93bd6e\" (UID: \"13089e4b-aad1-4aad-a869-1ef46d93bd6e\") " Nov 03 14:23:13.371661 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:13.371508 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13089e4b-aad1-4aad-a869-1ef46d93bd6e-tmp-dir" pod "13089e4b-aad1-4aad-a869-1ef46d93bd6e" (UID: "13089e4b-aad1-4aad-a869-1ef46d93bd6e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:23:13.372001 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:13.371796 5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13089e4b-aad1-4aad-a869-1ef46d93bd6e-tmp-dir\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:23:13.372001 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:13.371923 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13089e4b-aad1-4aad-a869-1ef46d93bd6e-model-cache" pod "13089e4b-aad1-4aad-a869-1ef46d93bd6e" (UID: "13089e4b-aad1-4aad-a869-1ef46d93bd6e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:23:13.372187 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:13.372167 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13089e4b-aad1-4aad-a869-1ef46d93bd6e-home" pod "13089e4b-aad1-4aad-a869-1ef46d93bd6e" (UID: "13089e4b-aad1-4aad-a869-1ef46d93bd6e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:23:13.374101 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:13.374073 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13089e4b-aad1-4aad-a869-1ef46d93bd6e-tls-certs" pod "13089e4b-aad1-4aad-a869-1ef46d93bd6e" (UID: "13089e4b-aad1-4aad-a869-1ef46d93bd6e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:23:13.374225 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:13.374205 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13089e4b-aad1-4aad-a869-1ef46d93bd6e-kube-api-access-2zxsl" pod "13089e4b-aad1-4aad-a869-1ef46d93bd6e" (UID: "13089e4b-aad1-4aad-a869-1ef46d93bd6e"). InnerVolumeSpecName "kube-api-access-2zxsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:23:13.374268 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:13.374239 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13089e4b-aad1-4aad-a869-1ef46d93bd6e-dshm" pod "13089e4b-aad1-4aad-a869-1ef46d93bd6e" (UID: "13089e4b-aad1-4aad-a869-1ef46d93bd6e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:23:13.473525 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:13.473491 5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13089e4b-aad1-4aad-a869-1ef46d93bd6e-dshm\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:23:13.473525 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:13.473522 5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13089e4b-aad1-4aad-a869-1ef46d93bd6e-tls-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:23:13.473525 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:13.473532 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zxsl\" (UniqueName: \"kubernetes.io/projected/13089e4b-aad1-4aad-a869-1ef46d93bd6e-kube-api-access-2zxsl\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:23:13.473947 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:13.473541 5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13089e4b-aad1-4aad-a869-1ef46d93bd6e-model-cache\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:23:13.473947 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:13.473549 5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13089e4b-aad1-4aad-a869-1ef46d93bd6e-home\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:23:14.153526 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:14.153496 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7316-83dcde"] Nov 03 14:23:14.159067 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:14.159040 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7316-83dcde"] Nov 03 14:23:14.272775 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:14.272741 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-4429aa/praxis-route-4429aa-kserve-668c498f79-4pb6w" Nov 03 14:23:14.274955 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:14.274918 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-4429aa/praxis-route-4429aa-kserve-668c498f79-4pb6w" event={"ID":"13089e4b-aad1-4aad-a869-1ef46d93bd6e","Type":"ContainerDied","Data":"c9492eeaf66c12c6590dff02252739ef67546e706101208951f054f6845f06f1"} Nov 03 14:23:14.275083 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:14.274981 5060 scope.go:122] "RemoveContainer" containerID="73874d214748f06bbb1041ae53874fefe47bf131c382d25fbcdd530d2c258f73" Nov 03 14:23:14.332343 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:14.332306 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-4429aa/praxis-route-4429aa-kserve-668c498f79-4pb6w"] Nov 03 14:23:14.343300 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:14.343262 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-4429aa/praxis-route-4429aa-kserve-668c498f79-4pb6w"] Nov 03 14:23:15.148821 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.148785 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6"] Nov 03 14:23:15.150206 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.150180 5060 cpu_manager.go:414] "containerMap: removing container" podUID="13089e4b-aad1-4aad-a869-1ef46d93bd6e" containerName="main" Nov 03 14:23:15.150374 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.150358 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13089e4b-aad1-4aad-a869-1ef46d93bd6e" containerName="main" Nov 03 14:23:15.150506 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.150494 5060 cpu_manager.go:414] "containerMap: removing container" podUID="8b440e41-f5ad-458d-a874-29d3a84eccf1" containerName="test-curl-7316-83dcde" Nov 03 14:23:15.150591 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.150580 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b440e41-f5ad-458d-a874-29d3a84eccf1" containerName="test-curl-7316-83dcde" Nov 03 14:23:15.150792 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.150780 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="13089e4b-aad1-4aad-a869-1ef46d93bd6e" containerName="main" Nov 03 14:23:15.150915 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.150902 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b440e41-f5ad-458d-a874-29d3a84eccf1" containerName="test-curl-7316-83dcde" Nov 03 14:23:15.226049 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.216286 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6"] Nov 03 14:23:15.226049 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.216518 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" Nov 03 14:23:15.244253 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.244220 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-5b97f1-openshift-default-dockercfg-gbzmt\"" Nov 03 14:23:15.290782 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.290746 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7316-83dcde" podUID="8b440e41-f5ad-458d-a874-29d3a84eccf1" containerName="test-curl-7316-83dcde" containerID="cri-o://9e3f532e5cf4f253e3fcdbe48c3346439b6a0e514c012232996c65c773332766" gracePeriod=2 Nov 03 14:23:15.399911 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.399406 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsb6p\" (UniqueName: \"kubernetes.io/projected/711e30e5-2c44-4295-974c-80a266623ae1-kube-api-access-fsb6p\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6\" (UID: \"711e30e5-2c44-4295-974c-80a266623ae1\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" Nov 03 14:23:15.399911 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.399466 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/711e30e5-2c44-4295-974c-80a266623ae1-workload-certs\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6\" (UID: \"711e30e5-2c44-4295-974c-80a266623ae1\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" Nov 03 14:23:15.399911 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.399499 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/711e30e5-2c44-4295-974c-80a266623ae1-istio-data\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6\" (UID: \"711e30e5-2c44-4295-974c-80a266623ae1\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" Nov 03 14:23:15.399911 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.399535 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/711e30e5-2c44-4295-974c-80a266623ae1-credential-socket\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6\" (UID: \"711e30e5-2c44-4295-974c-80a266623ae1\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" Nov 03 14:23:15.399911 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.399592 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/711e30e5-2c44-4295-974c-80a266623ae1-workload-socket\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6\" (UID: \"711e30e5-2c44-4295-974c-80a266623ae1\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" Nov 03 14:23:15.399911 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.399621 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/711e30e5-2c44-4295-974c-80a266623ae1-istio-token\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6\" (UID: \"711e30e5-2c44-4295-974c-80a266623ae1\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" Nov 03 14:23:15.399911 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.399697 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/711e30e5-2c44-4295-974c-80a266623ae1-istiod-ca-cert\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6\" (UID: \"711e30e5-2c44-4295-974c-80a266623ae1\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" Nov 03 14:23:15.399911 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.399763 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/711e30e5-2c44-4295-974c-80a266623ae1-istio-podinfo\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6\" (UID: \"711e30e5-2c44-4295-974c-80a266623ae1\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" Nov 03 14:23:15.399911 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.399810 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/711e30e5-2c44-4295-974c-80a266623ae1-istio-envoy\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6\" (UID: \"711e30e5-2c44-4295-974c-80a266623ae1\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" Nov 03 14:23:15.500723 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.500682 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/711e30e5-2c44-4295-974c-80a266623ae1-credential-socket\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6\" (UID: \"711e30e5-2c44-4295-974c-80a266623ae1\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" Nov 03 14:23:15.500949 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.500744 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/711e30e5-2c44-4295-974c-80a266623ae1-workload-socket\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6\" (UID: \"711e30e5-2c44-4295-974c-80a266623ae1\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" Nov 03 14:23:15.500949 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.500772 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/711e30e5-2c44-4295-974c-80a266623ae1-istio-token\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6\" (UID: \"711e30e5-2c44-4295-974c-80a266623ae1\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" Nov 03 14:23:15.500949 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.500850 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/711e30e5-2c44-4295-974c-80a266623ae1-istiod-ca-cert\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6\" (UID: \"711e30e5-2c44-4295-974c-80a266623ae1\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" Nov 03 14:23:15.500949 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.500904 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/711e30e5-2c44-4295-974c-80a266623ae1-istio-podinfo\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6\" (UID: \"711e30e5-2c44-4295-974c-80a266623ae1\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" Nov 03 14:23:15.500949 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.500942 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/711e30e5-2c44-4295-974c-80a266623ae1-istio-envoy\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6\" (UID: \"711e30e5-2c44-4295-974c-80a266623ae1\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" Nov 03 14:23:15.501233 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.500996 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsb6p\" (UniqueName: \"kubernetes.io/projected/711e30e5-2c44-4295-974c-80a266623ae1-kube-api-access-fsb6p\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6\" (UID: \"711e30e5-2c44-4295-974c-80a266623ae1\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" Nov 03 14:23:15.501233 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.501027 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/711e30e5-2c44-4295-974c-80a266623ae1-workload-certs\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6\" (UID: \"711e30e5-2c44-4295-974c-80a266623ae1\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" Nov 03 14:23:15.501233 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.501055 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/711e30e5-2c44-4295-974c-80a266623ae1-istio-data\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6\" (UID: \"711e30e5-2c44-4295-974c-80a266623ae1\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" Nov 03 14:23:15.501233 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.501167 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/711e30e5-2c44-4295-974c-80a266623ae1-credential-socket\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6\" (UID: \"711e30e5-2c44-4295-974c-80a266623ae1\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" Nov 03 14:23:15.501233 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.501212 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/711e30e5-2c44-4295-974c-80a266623ae1-workload-socket\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6\" (UID: \"711e30e5-2c44-4295-974c-80a266623ae1\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" Nov 03 14:23:15.501500 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.501433 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/711e30e5-2c44-4295-974c-80a266623ae1-istio-data\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6\" (UID: \"711e30e5-2c44-4295-974c-80a266623ae1\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" Nov 03 14:23:15.501500 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.501478 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/711e30e5-2c44-4295-974c-80a266623ae1-workload-certs\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6\" (UID: \"711e30e5-2c44-4295-974c-80a266623ae1\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" Nov 03 14:23:15.501885 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.501858 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/711e30e5-2c44-4295-974c-80a266623ae1-istiod-ca-cert\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6\" (UID: \"711e30e5-2c44-4295-974c-80a266623ae1\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" Nov 03 14:23:15.503593 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.503558 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/711e30e5-2c44-4295-974c-80a266623ae1-istio-envoy\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6\" (UID: \"711e30e5-2c44-4295-974c-80a266623ae1\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" Nov 03 14:23:15.503905 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.503884 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/711e30e5-2c44-4295-974c-80a266623ae1-istio-podinfo\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6\" (UID: \"711e30e5-2c44-4295-974c-80a266623ae1\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" Nov 03 14:23:15.526348 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.526319 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsb6p\" (UniqueName: \"kubernetes.io/projected/711e30e5-2c44-4295-974c-80a266623ae1-kube-api-access-fsb6p\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6\" (UID: \"711e30e5-2c44-4295-974c-80a266623ae1\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" Nov 03 14:23:15.526777 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.526755 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/711e30e5-2c44-4295-974c-80a266623ae1-istio-token\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6\" (UID: \"711e30e5-2c44-4295-974c-80a266623ae1\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" Nov 03 14:23:15.549779 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.549750 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7316-4069cb" Nov 03 14:23:15.579612 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.579575 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" Nov 03 14:23:15.601791 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.601766 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71d5e661-5675-4b1c-b1e5-24a3c65829e0-kube-api-access-ztgvh\" (UniqueName: \"kubernetes.io/projected/71d5e661-5675-4b1c-b1e5-24a3c65829e0-kube-api-access-ztgvh\") pod \"71d5e661-5675-4b1c-b1e5-24a3c65829e0\" (UID: \"71d5e661-5675-4b1c-b1e5-24a3c65829e0\") " Nov 03 14:23:15.603783 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.603758 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d5e661-5675-4b1c-b1e5-24a3c65829e0-kube-api-access-ztgvh" pod "71d5e661-5675-4b1c-b1e5-24a3c65829e0" (UID: "71d5e661-5675-4b1c-b1e5-24a3c65829e0"). InnerVolumeSpecName "kube-api-access-ztgvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:23:15.703003 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.702906 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztgvh\" (UniqueName: \"kubernetes.io/projected/71d5e661-5675-4b1c-b1e5-24a3c65829e0-kube-api-access-ztgvh\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:23:15.908808 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.908770 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6"] Nov 03 14:23:15.912616 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:23:15.912584 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod711e30e5_2c44_4295_974c_80a266623ae1.slice/crio-996b2b17a76d765c4dbcd2207117b1cab8978f625a9388ce991d0e725ddd0c66 WatchSource:0}: Error finding container 996b2b17a76d765c4dbcd2207117b1cab8978f625a9388ce991d0e725ddd0c66: Status 404 returned error can't find the container with id 996b2b17a76d765c4dbcd2207117b1cab8978f625a9388ce991d0e725ddd0c66 Nov 03 14:23:15.915617 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.915576 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:23:15.915711 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.915662 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:23:15.915745 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:15.915733 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:23:16.250060 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:16.250025 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13089e4b-aad1-4aad-a869-1ef46d93bd6e" path="/var/lib/kubelet/pods/13089e4b-aad1-4aad-a869-1ef46d93bd6e/volumes" Nov 03 14:23:16.250547 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:16.250526 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71d5e661-5675-4b1c-b1e5-24a3c65829e0" path="/var/lib/kubelet/pods/71d5e661-5675-4b1c-b1e5-24a3c65829e0/volumes" Nov 03 14:23:16.298305 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:16.298267 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" event={"ID":"711e30e5-2c44-4295-974c-80a266623ae1","Type":"ContainerStarted","Data":"0ba1c7f2a6096b1f7b4f7d4f5e68ed118f027a5f9427f6aa40d4ed459eba6fab"} Nov 03 14:23:16.298428 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:16.298336 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" event={"ID":"711e30e5-2c44-4295-974c-80a266623ae1","Type":"ContainerStarted","Data":"996b2b17a76d765c4dbcd2207117b1cab8978f625a9388ce991d0e725ddd0c66"} Nov 03 14:23:16.299984 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:16.299942 5060 generic.go:356] "Generic (PLEG): container finished" podID="71d5e661-5675-4b1c-b1e5-24a3c65829e0" containerID="7709861185af960338ad14449398d400972ba748a237d9eefdf29370670ade14" exitCode=137 Nov 03 14:23:16.300114 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:16.299994 5060 scope.go:122] "RemoveContainer" containerID="7709861185af960338ad14449398d400972ba748a237d9eefdf29370670ade14" Nov 03 14:23:16.300114 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:16.299968 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7316-4069cb" Nov 03 14:23:16.310448 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:16.310421 5060 scope.go:122] "RemoveContainer" containerID="7709861185af960338ad14449398d400972ba748a237d9eefdf29370670ade14" Nov 03 14:23:16.310733 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:23:16.310710 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7709861185af960338ad14449398d400972ba748a237d9eefdf29370670ade14\": container with ID starting with 7709861185af960338ad14449398d400972ba748a237d9eefdf29370670ade14 not found: ID does not exist" containerID="7709861185af960338ad14449398d400972ba748a237d9eefdf29370670ade14" Nov 03 14:23:16.310790 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:16.310744 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7709861185af960338ad14449398d400972ba748a237d9eefdf29370670ade14"} err="failed to get container status \"7709861185af960338ad14449398d400972ba748a237d9eefdf29370670ade14\": rpc error: code = NotFound desc = could not find container \"7709861185af960338ad14449398d400972ba748a237d9eefdf29370670ade14\": container with ID starting with 7709861185af960338ad14449398d400972ba748a237d9eefdf29370670ade14 not found: ID does not exist" Nov 03 14:23:16.344682 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:16.344617 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" podStartSLOduration=1.344603183 podStartE2EDuration="1.344603183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:23:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:23:16.34327818 +0000 UTC m=+730.232067773" watchObservedRunningTime="2026-11-03 14:23:16.344603183 +0000 UTC m=+730.233392764" Nov 03 14:23:16.579811 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:16.579726 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" Nov 03 14:23:16.585805 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:16.585781 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" Nov 03 14:23:17.307549 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:17.307521 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" Nov 03 14:23:17.308632 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:17.308606 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" Nov 03 14:23:17.460372 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:17.460326 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr" podUID="94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4" containerName="main" containerID="cri-o://90544d737dbfc552c79a3358ab4618fa11277154074aa43b1170ef2919d1175f" gracePeriod=45 Nov 03 14:23:17.590769 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:17.590738 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7316-83dcde" Nov 03 14:23:17.621714 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:17.621679 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b440e41-f5ad-458d-a874-29d3a84eccf1-kube-api-access-gczcm\" (UniqueName: \"kubernetes.io/projected/8b440e41-f5ad-458d-a874-29d3a84eccf1-kube-api-access-gczcm\") pod \"8b440e41-f5ad-458d-a874-29d3a84eccf1\" (UID: \"8b440e41-f5ad-458d-a874-29d3a84eccf1\") " Nov 03 14:23:17.623731 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:17.623704 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b440e41-f5ad-458d-a874-29d3a84eccf1-kube-api-access-gczcm" pod "8b440e41-f5ad-458d-a874-29d3a84eccf1" (UID: "8b440e41-f5ad-458d-a874-29d3a84eccf1"). InnerVolumeSpecName "kube-api-access-gczcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:23:17.718007 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:17.717981 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr" Nov 03 14:23:17.723074 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:17.723045 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gczcm\" (UniqueName: \"kubernetes.io/projected/8b440e41-f5ad-458d-a874-29d3a84eccf1-kube-api-access-gczcm\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:23:17.823750 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:17.823659 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4-home\" (UniqueName: \"kubernetes.io/empty-dir/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4-home\") pod \"94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4\" (UID: \"94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4\") " Nov 03 14:23:17.823750 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:17.823740 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4-dshm\") pod \"94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4\" (UID: \"94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4\") " Nov 03 14:23:17.824008 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:17.823780 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4-tls-certs\" (UniqueName: \"kubernetes.io/secret/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4-tls-certs\") pod \"94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4\" (UID: \"94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4\") " Nov 03 14:23:17.824008 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:17.823825 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4-model-cache\") pod \"94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4\" (UID: \"94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4\") " Nov 03 14:23:17.824008 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:17.823972 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4-tmp-dir\") pod \"94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4\" (UID: \"94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4\") " Nov 03 14:23:17.824008 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:17.823986 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4-home" pod "94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4" (UID: "94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:23:17.824208 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:17.824021 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4-kube-api-access-hn5b2\" (UniqueName: \"kubernetes.io/projected/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4-kube-api-access-hn5b2\") pod \"94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4\" (UID: \"94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4\") " Nov 03 14:23:17.824208 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:17.824179 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4-model-cache" pod "94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4" (UID: "94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:23:17.824208 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:17.824200 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4-tmp-dir" pod "94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4" (UID: "94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:23:17.824367 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:17.824359 5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4-model-cache\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:23:17.824418 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:17.824374 5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4-tmp-dir\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:23:17.824418 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:17.824387 5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4-home\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:23:17.826290 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:17.826263 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4-kube-api-access-hn5b2" pod "94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4" (UID: "94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4"). InnerVolumeSpecName "kube-api-access-hn5b2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:23:17.826438 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:17.826420 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4-dshm" pod "94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4" (UID: "94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:23:17.826493 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:17.826427 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4-tls-certs" pod "94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4" (UID: "94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:23:17.925279 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:17.925242 5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4-dshm\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:23:17.925279 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:17.925270 5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4-tls-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:23:17.925279 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:17.925283 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn5b2\" (UniqueName: \"kubernetes.io/projected/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4-kube-api-access-hn5b2\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:23:18.159643 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:18.159554 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98"] Nov 03 14:23:18.160138 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:18.160118 5060 cpu_manager.go:414] "containerMap: removing container" podUID="94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4" containerName="main" Nov 03 14:23:18.160252 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:18.160139 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4" containerName="main" Nov 03 14:23:18.160252 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:18.160244 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4" containerName="main" Nov 03 14:23:18.163327 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:18.163300 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98" Nov 03 14:23:18.179503 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:18.179482 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-156ae1\"/\"distinct-w3-156ae1-kserve-self-signed-certs\"" Nov 03 14:23:18.193720 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:18.193688 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98"] Nov 03 14:23:18.228958 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:18.228924 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3aaaeea4-134f-4d17-a180-b4857c6425d3-dshm\") pod \"distinct-w3-156ae1-kserve-865cd6c59f-8rc98\" (UID: \"3aaaeea4-134f-4d17-a180-b4857c6425d3\") " pod="e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98" Nov 03 14:23:18.229149 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:18.228967 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrcf8\" (UniqueName: \"kubernetes.io/projected/3aaaeea4-134f-4d17-a180-b4857c6425d3-kube-api-access-lrcf8\") pod \"distinct-w3-156ae1-kserve-865cd6c59f-8rc98\" (UID: \"3aaaeea4-134f-4d17-a180-b4857c6425d3\") " pod="e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98" Nov 03 14:23:18.229149 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:18.228997 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3aaaeea4-134f-4d17-a180-b4857c6425d3-home\") pod \"distinct-w3-156ae1-kserve-865cd6c59f-8rc98\" (UID: \"3aaaeea4-134f-4d17-a180-b4857c6425d3\") " pod="e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98" Nov 03 14:23:18.229149 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:18.229114 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3aaaeea4-134f-4d17-a180-b4857c6425d3-model-cache\") pod \"distinct-w3-156ae1-kserve-865cd6c59f-8rc98\" (UID: \"3aaaeea4-134f-4d17-a180-b4857c6425d3\") " pod="e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98" Nov 03 14:23:18.229331 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:18.229197 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3aaaeea4-134f-4d17-a180-b4857c6425d3-tmp-dir\") pod \"distinct-w3-156ae1-kserve-865cd6c59f-8rc98\" (UID: \"3aaaeea4-134f-4d17-a180-b4857c6425d3\") " pod="e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98" Nov 03 14:23:18.229331 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:18.229262 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3aaaeea4-134f-4d17-a180-b4857c6425d3-tls-certs\") pod \"distinct-w3-156ae1-kserve-865cd6c59f-8rc98\" (UID: \"3aaaeea4-134f-4d17-a180-b4857c6425d3\") " pod="e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98" Nov 03 14:23:18.250582 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:18.250545 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b440e41-f5ad-458d-a874-29d3a84eccf1" path="/var/lib/kubelet/pods/8b440e41-f5ad-458d-a874-29d3a84eccf1/volumes" Nov 03 14:23:18.315151 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:18.315120 5060 generic.go:356] "Generic (PLEG): container finished" podID="8b440e41-f5ad-458d-a874-29d3a84eccf1" containerID="9e3f532e5cf4f253e3fcdbe48c3346439b6a0e514c012232996c65c773332766" exitCode=137 Nov 03 14:23:18.315574 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:18.315182 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7316-83dcde" Nov 03 14:23:18.315574 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:18.315216 5060 scope.go:122] "RemoveContainer" containerID="9e3f532e5cf4f253e3fcdbe48c3346439b6a0e514c012232996c65c773332766" Nov 03 14:23:18.316989 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:18.316965 5060 generic.go:356] "Generic (PLEG): container finished" podID="94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4" containerID="90544d737dbfc552c79a3358ab4618fa11277154074aa43b1170ef2919d1175f" exitCode=0 Nov 03 14:23:18.317117 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:18.317027 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr" Nov 03 14:23:18.317117 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:18.317035 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr" event={"ID":"94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4","Type":"ContainerDied","Data":"90544d737dbfc552c79a3358ab4618fa11277154074aa43b1170ef2919d1175f"} Nov 03 14:23:18.317117 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:18.317101 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-156ae1/e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr" event={"ID":"94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4","Type":"ContainerDied","Data":"9c6662d926cdd5d5ccf698a30ea7ee5df7ee3866e8a30b6185c2eaf86aad179d"} Nov 03 14:23:18.325727 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:18.325710 5060 scope.go:122] "RemoveContainer" containerID="9e3f532e5cf4f253e3fcdbe48c3346439b6a0e514c012232996c65c773332766" Nov 03 14:23:18.326077 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:23:18.326052 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e3f532e5cf4f253e3fcdbe48c3346439b6a0e514c012232996c65c773332766\": container with ID starting with 9e3f532e5cf4f253e3fcdbe48c3346439b6a0e514c012232996c65c773332766 not found: ID does not exist" containerID="9e3f532e5cf4f253e3fcdbe48c3346439b6a0e514c012232996c65c773332766" Nov 03 14:23:18.326172 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:18.326087 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e3f532e5cf4f253e3fcdbe48c3346439b6a0e514c012232996c65c773332766"} err="failed to get container status \"9e3f532e5cf4f253e3fcdbe48c3346439b6a0e514c012232996c65c773332766\": rpc error: code = NotFound desc = could not find container \"9e3f532e5cf4f253e3fcdbe48c3346439b6a0e514c012232996c65c773332766\": container with ID starting with 9e3f532e5cf4f253e3fcdbe48c3346439b6a0e514c012232996c65c773332766 not found: ID does not exist" Nov 03 14:23:18.326172 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:18.326108 5060 scope.go:122] "RemoveContainer" containerID="90544d737dbfc552c79a3358ab4618fa11277154074aa43b1170ef2919d1175f" Nov 03 14:23:18.330688 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:18.330661 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3aaaeea4-134f-4d17-a180-b4857c6425d3-dshm\") pod \"distinct-w3-156ae1-kserve-865cd6c59f-8rc98\" (UID: \"3aaaeea4-134f-4d17-a180-b4857c6425d3\") " pod="e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98" Nov 03 14:23:18.330779 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:18.330710 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrcf8\" (UniqueName: \"kubernetes.io/projected/3aaaeea4-134f-4d17-a180-b4857c6425d3-kube-api-access-lrcf8\") pod \"distinct-w3-156ae1-kserve-865cd6c59f-8rc98\" (UID: \"3aaaeea4-134f-4d17-a180-b4857c6425d3\") " pod="e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98" Nov 03 14:23:18.330779 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:18.330770 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3aaaeea4-134f-4d17-a180-b4857c6425d3-home\") pod \"distinct-w3-156ae1-kserve-865cd6c59f-8rc98\" (UID: \"3aaaeea4-134f-4d17-a180-b4857c6425d3\") " pod="e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98" Nov 03 14:23:18.330900 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:18.330800 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3aaaeea4-134f-4d17-a180-b4857c6425d3-model-cache\") pod \"distinct-w3-156ae1-kserve-865cd6c59f-8rc98\" (UID: \"3aaaeea4-134f-4d17-a180-b4857c6425d3\") " pod="e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98" Nov 03 14:23:18.330900 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:18.330889 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3aaaeea4-134f-4d17-a180-b4857c6425d3-tmp-dir\") pod \"distinct-w3-156ae1-kserve-865cd6c59f-8rc98\" (UID: \"3aaaeea4-134f-4d17-a180-b4857c6425d3\") " pod="e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98" Nov 03 14:23:18.331011 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:18.330930 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3aaaeea4-134f-4d17-a180-b4857c6425d3-tls-certs\") pod \"distinct-w3-156ae1-kserve-865cd6c59f-8rc98\" (UID: \"3aaaeea4-134f-4d17-a180-b4857c6425d3\") " pod="e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98" Nov 03 14:23:18.331702 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:18.331394 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3aaaeea4-134f-4d17-a180-b4857c6425d3-model-cache\") pod \"distinct-w3-156ae1-kserve-865cd6c59f-8rc98\" (UID: \"3aaaeea4-134f-4d17-a180-b4857c6425d3\") " pod="e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98" Nov 03 14:23:18.331702 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:18.331460 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3aaaeea4-134f-4d17-a180-b4857c6425d3-home\") pod \"distinct-w3-156ae1-kserve-865cd6c59f-8rc98\" (UID: \"3aaaeea4-134f-4d17-a180-b4857c6425d3\") " pod="e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98" Nov 03 14:23:18.331702 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:18.331664 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3aaaeea4-134f-4d17-a180-b4857c6425d3-tmp-dir\") pod \"distinct-w3-156ae1-kserve-865cd6c59f-8rc98\" (UID: \"3aaaeea4-134f-4d17-a180-b4857c6425d3\") " pod="e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98" Nov 03 14:23:18.333492 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:18.333473 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3aaaeea4-134f-4d17-a180-b4857c6425d3-dshm\") pod \"distinct-w3-156ae1-kserve-865cd6c59f-8rc98\" (UID: \"3aaaeea4-134f-4d17-a180-b4857c6425d3\") " pod="e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98" Nov 03 14:23:18.334155 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:18.334137 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3aaaeea4-134f-4d17-a180-b4857c6425d3-tls-certs\") pod \"distinct-w3-156ae1-kserve-865cd6c59f-8rc98\" (UID: \"3aaaeea4-134f-4d17-a180-b4857c6425d3\") " pod="e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98" Nov 03 14:23:18.337247 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:18.337121 5060 scope.go:122] "RemoveContainer" containerID="90544d737dbfc552c79a3358ab4618fa11277154074aa43b1170ef2919d1175f" Nov 03 14:23:18.337452 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:23:18.337431 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90544d737dbfc552c79a3358ab4618fa11277154074aa43b1170ef2919d1175f\": container with ID starting with 90544d737dbfc552c79a3358ab4618fa11277154074aa43b1170ef2919d1175f not found: ID does not exist" containerID="90544d737dbfc552c79a3358ab4618fa11277154074aa43b1170ef2919d1175f" Nov 03 14:23:18.337509 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:18.337457 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90544d737dbfc552c79a3358ab4618fa11277154074aa43b1170ef2919d1175f"} err="failed to get container status \"90544d737dbfc552c79a3358ab4618fa11277154074aa43b1170ef2919d1175f\": rpc error: code = NotFound desc = could not find container \"90544d737dbfc552c79a3358ab4618fa11277154074aa43b1170ef2919d1175f\": container with ID starting with 90544d737dbfc552c79a3358ab4618fa11277154074aa43b1170ef2919d1175f not found: ID does not exist" Nov 03 14:23:18.356644 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:18.352359 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrcf8\" (UniqueName: \"kubernetes.io/projected/3aaaeea4-134f-4d17-a180-b4857c6425d3-kube-api-access-lrcf8\") pod \"distinct-w3-156ae1-kserve-865cd6c59f-8rc98\" (UID: \"3aaaeea4-134f-4d17-a180-b4857c6425d3\") " pod="e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98" Nov 03 14:23:18.391970 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:18.391931 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-156ae1/e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr"] Nov 03 14:23:18.393440 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:18.393404 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-156ae1/e2e-conflict-b-1c60a22c-kserve-798dc45d75-nrdpr"] Nov 03 14:23:18.478807 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:18.478715 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98" Nov 03 14:23:18.787191 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:18.787158 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98"] Nov 03 14:23:18.789378 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:23:18.789354 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3aaaeea4_134f_4d17_a180_b4857c6425d3.slice/crio-4581efed9093a0967c17779cde329fd164ca71bace7efb07be66a871cdec0362 WatchSource:0}: Error finding container 4581efed9093a0967c17779cde329fd164ca71bace7efb07be66a871cdec0362: Status 404 returned error can't find the container with id 4581efed9093a0967c17779cde329fd164ca71bace7efb07be66a871cdec0362 Nov 03 14:23:19.324080 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:19.324041 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98" event={"ID":"3aaaeea4-134f-4d17-a180-b4857c6425d3","Type":"ContainerStarted","Data":"ea006e5bfb0cf3fca5446ad0f429c1bcc9b3d0e0c98c79459a13e68ecd12bbc6"} Nov 03 14:23:19.324516 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:19.324093 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98" event={"ID":"3aaaeea4-134f-4d17-a180-b4857c6425d3","Type":"ContainerStarted","Data":"4581efed9093a0967c17779cde329fd164ca71bace7efb07be66a871cdec0362"} Nov 03 14:23:19.367372 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:19.364937 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98" podStartSLOduration=1.364924081 podStartE2EDuration="1.364924081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:23:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:23:19.364042566 +0000 UTC m=+733.252832148" watchObservedRunningTime="2026-11-03 14:23:19.364924081 +0000 UTC m=+733.253713662" Nov 03 14:23:19.479635 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:19.479599 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98" Nov 03 14:23:20.248884 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:20.248829 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4" path="/var/lib/kubelet/pods/94ccad46-ebe5-4a0b-9cff-6efaf6ba66b4/volumes" Nov 03 14:23:24.711734 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:24.711704 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-041fbab7-4pp7d"] Nov 03 14:23:24.715602 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:24.715583 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-041fbab7-4pp7d" Nov 03 14:23:24.741932 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:24.741905 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\"" Nov 03 14:23:24.742104 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:24.741915 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:23:24.742104 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:24.741972 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:23:24.742256 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:24.742004 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:23:24.744904 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:24.744880 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-041fbab7-4pp7d"] Nov 03 14:23:24.795463 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:24.795435 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcmd7\" (UniqueName: \"kubernetes.io/projected/683f2838-8f53-4f9e-8bb5-a156ca3ff2d8-kube-api-access-rcmd7\") pod \"maas-api-revoke-sub-041fbab7-4pp7d\" (UID: \"683f2838-8f53-4f9e-8bb5-a156ca3ff2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-041fbab7-4pp7d" Nov 03 14:23:24.795602 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:24.795499 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/683f2838-8f53-4f9e-8bb5-a156ca3ff2d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-041fbab7-4pp7d\" (UID: \"683f2838-8f53-4f9e-8bb5-a156ca3ff2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-041fbab7-4pp7d" Nov 03 14:23:24.795602 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:24.795519 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/683f2838-8f53-4f9e-8bb5-a156ca3ff2d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-041fbab7-4pp7d\" (UID: \"683f2838-8f53-4f9e-8bb5-a156ca3ff2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-041fbab7-4pp7d" Nov 03 14:23:24.896655 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:24.896616 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcmd7\" (UniqueName: \"kubernetes.io/projected/683f2838-8f53-4f9e-8bb5-a156ca3ff2d8-kube-api-access-rcmd7\") pod \"maas-api-revoke-sub-041fbab7-4pp7d\" (UID: \"683f2838-8f53-4f9e-8bb5-a156ca3ff2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-041fbab7-4pp7d" Nov 03 14:23:24.896811 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:24.896692 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/683f2838-8f53-4f9e-8bb5-a156ca3ff2d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-041fbab7-4pp7d\" (UID: \"683f2838-8f53-4f9e-8bb5-a156ca3ff2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-041fbab7-4pp7d" Nov 03 14:23:24.896811 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:24.896714 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/683f2838-8f53-4f9e-8bb5-a156ca3ff2d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-041fbab7-4pp7d\" (UID: \"683f2838-8f53-4f9e-8bb5-a156ca3ff2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-041fbab7-4pp7d" Nov 03 14:23:24.897559 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:24.897531 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/683f2838-8f53-4f9e-8bb5-a156ca3ff2d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-041fbab7-4pp7d\" (UID: \"683f2838-8f53-4f9e-8bb5-a156ca3ff2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-041fbab7-4pp7d" Nov 03 14:23:24.899080 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:24.899049 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/683f2838-8f53-4f9e-8bb5-a156ca3ff2d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-041fbab7-4pp7d\" (UID: \"683f2838-8f53-4f9e-8bb5-a156ca3ff2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-041fbab7-4pp7d" Nov 03 14:23:24.917290 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:24.917261 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcmd7\" (UniqueName: \"kubernetes.io/projected/683f2838-8f53-4f9e-8bb5-a156ca3ff2d8-kube-api-access-rcmd7\") pod \"maas-api-revoke-sub-041fbab7-4pp7d\" (UID: \"683f2838-8f53-4f9e-8bb5-a156ca3ff2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-041fbab7-4pp7d" Nov 03 14:23:25.029507 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.029467 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-041fbab7-4pp7d" Nov 03 14:23:25.103661 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.103624 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5"] Nov 03 14:23:25.108479 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.108455 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" Nov 03 14:23:25.135958 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.135881 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5"] Nov 03 14:23:25.199669 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.199629 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8172c42-332c-40e4-9317-9014c5749253-istio-token\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5\" (UID: \"c8172c42-332c-40e4-9317-9014c5749253\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" Nov 03 14:23:25.199880 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.199699 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8172c42-332c-40e4-9317-9014c5749253-istiod-ca-cert\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5\" (UID: \"c8172c42-332c-40e4-9317-9014c5749253\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" Nov 03 14:23:25.199880 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.199807 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8172c42-332c-40e4-9317-9014c5749253-workload-certs\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5\" (UID: \"c8172c42-332c-40e4-9317-9014c5749253\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" Nov 03 14:23:25.199880 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.199864 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8172c42-332c-40e4-9317-9014c5749253-istio-podinfo\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5\" (UID: \"c8172c42-332c-40e4-9317-9014c5749253\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" Nov 03 14:23:25.200000 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.199927 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8172c42-332c-40e4-9317-9014c5749253-workload-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5\" (UID: \"c8172c42-332c-40e4-9317-9014c5749253\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" Nov 03 14:23:25.200038 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.199999 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8172c42-332c-40e4-9317-9014c5749253-credential-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5\" (UID: \"c8172c42-332c-40e4-9317-9014c5749253\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" Nov 03 14:23:25.200038 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.200026 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8172c42-332c-40e4-9317-9014c5749253-istio-data\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5\" (UID: \"c8172c42-332c-40e4-9317-9014c5749253\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" Nov 03 14:23:25.200101 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.200057 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8172c42-332c-40e4-9317-9014c5749253-istio-envoy\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5\" (UID: \"c8172c42-332c-40e4-9317-9014c5749253\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" Nov 03 14:23:25.200184 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.200159 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz2c9\" (UniqueName: \"kubernetes.io/projected/c8172c42-332c-40e4-9317-9014c5749253-kube-api-access-dz2c9\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5\" (UID: \"c8172c42-332c-40e4-9317-9014c5749253\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" Nov 03 14:23:25.266766 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.266736 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8"] Nov 03 14:23:25.270531 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.270511 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" Nov 03 14:23:25.294576 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.294501 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8"] Nov 03 14:23:25.301207 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.301165 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz2c9\" (UniqueName: \"kubernetes.io/projected/c8172c42-332c-40e4-9317-9014c5749253-kube-api-access-dz2c9\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5\" (UID: \"c8172c42-332c-40e4-9317-9014c5749253\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" Nov 03 14:23:25.301342 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.301231 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a88d21ce-086a-414a-911a-5561e2cb8023-istiod-ca-cert\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8\" (UID: \"a88d21ce-086a-414a-911a-5561e2cb8023\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" Nov 03 14:23:25.301342 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.301274 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8172c42-332c-40e4-9317-9014c5749253-istio-token\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5\" (UID: \"c8172c42-332c-40e4-9317-9014c5749253\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" Nov 03 14:23:25.301342 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.301303 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a88d21ce-086a-414a-911a-5561e2cb8023-istio-token\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8\" (UID: \"a88d21ce-086a-414a-911a-5561e2cb8023\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" Nov 03 14:23:25.301712 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.301376 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8172c42-332c-40e4-9317-9014c5749253-istiod-ca-cert\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5\" (UID: \"c8172c42-332c-40e4-9317-9014c5749253\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" Nov 03 14:23:25.301712 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.301438 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a88d21ce-086a-414a-911a-5561e2cb8023-istio-data\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8\" (UID: \"a88d21ce-086a-414a-911a-5561e2cb8023\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" Nov 03 14:23:25.301712 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.301481 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a88d21ce-086a-414a-911a-5561e2cb8023-workload-certs\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8\" (UID: \"a88d21ce-086a-414a-911a-5561e2cb8023\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" Nov 03 14:23:25.301712 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.301608 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tms5j\" (UniqueName: \"kubernetes.io/projected/a88d21ce-086a-414a-911a-5561e2cb8023-kube-api-access-tms5j\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8\" (UID: \"a88d21ce-086a-414a-911a-5561e2cb8023\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" Nov 03 14:23:25.301712 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.301648 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8172c42-332c-40e4-9317-9014c5749253-workload-certs\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5\" (UID: \"c8172c42-332c-40e4-9317-9014c5749253\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" Nov 03 14:23:25.301712 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.301676 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8172c42-332c-40e4-9317-9014c5749253-istio-podinfo\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5\" (UID: \"c8172c42-332c-40e4-9317-9014c5749253\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" Nov 03 14:23:25.301712 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.301707 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a88d21ce-086a-414a-911a-5561e2cb8023-workload-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8\" (UID: \"a88d21ce-086a-414a-911a-5561e2cb8023\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" Nov 03 14:23:25.302177 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.301734 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a88d21ce-086a-414a-911a-5561e2cb8023-credential-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8\" (UID: \"a88d21ce-086a-414a-911a-5561e2cb8023\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" Nov 03 14:23:25.302177 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.301775 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8172c42-332c-40e4-9317-9014c5749253-workload-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5\" (UID: \"c8172c42-332c-40e4-9317-9014c5749253\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" Nov 03 14:23:25.302177 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.301858 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8172c42-332c-40e4-9317-9014c5749253-credential-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5\" (UID: \"c8172c42-332c-40e4-9317-9014c5749253\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" Nov 03 14:23:25.302177 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.301884 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8172c42-332c-40e4-9317-9014c5749253-istio-data\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5\" (UID: \"c8172c42-332c-40e4-9317-9014c5749253\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" Nov 03 14:23:25.302177 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.301921 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8172c42-332c-40e4-9317-9014c5749253-istio-envoy\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5\" (UID: \"c8172c42-332c-40e4-9317-9014c5749253\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" Nov 03 14:23:25.302177 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.301969 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a88d21ce-086a-414a-911a-5561e2cb8023-istio-podinfo\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8\" (UID: \"a88d21ce-086a-414a-911a-5561e2cb8023\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" Nov 03 14:23:25.302177 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.301997 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a88d21ce-086a-414a-911a-5561e2cb8023-istio-envoy\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8\" (UID: \"a88d21ce-086a-414a-911a-5561e2cb8023\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" Nov 03 14:23:25.302540 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.302396 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8172c42-332c-40e4-9317-9014c5749253-workload-certs\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5\" (UID: \"c8172c42-332c-40e4-9317-9014c5749253\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" Nov 03 14:23:25.302540 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.302489 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8172c42-332c-40e4-9317-9014c5749253-istiod-ca-cert\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5\" (UID: \"c8172c42-332c-40e4-9317-9014c5749253\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" Nov 03 14:23:25.303123 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.303096 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8172c42-332c-40e4-9317-9014c5749253-credential-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5\" (UID: \"c8172c42-332c-40e4-9317-9014c5749253\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" Nov 03 14:23:25.303340 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.303318 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8172c42-332c-40e4-9317-9014c5749253-workload-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5\" (UID: \"c8172c42-332c-40e4-9317-9014c5749253\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" Nov 03 14:23:25.305133 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.305106 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8172c42-332c-40e4-9317-9014c5749253-istio-data\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5\" (UID: \"c8172c42-332c-40e4-9317-9014c5749253\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" Nov 03 14:23:25.305417 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.305395 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8172c42-332c-40e4-9317-9014c5749253-istio-podinfo\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5\" (UID: \"c8172c42-332c-40e4-9317-9014c5749253\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" Nov 03 14:23:25.305510 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.305453 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8172c42-332c-40e4-9317-9014c5749253-istio-envoy\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5\" (UID: \"c8172c42-332c-40e4-9317-9014c5749253\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" Nov 03 14:23:25.320946 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.320925 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8172c42-332c-40e4-9317-9014c5749253-istio-token\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5\" (UID: \"c8172c42-332c-40e4-9317-9014c5749253\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" Nov 03 14:23:25.321192 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.321150 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz2c9\" (UniqueName: \"kubernetes.io/projected/c8172c42-332c-40e4-9317-9014c5749253-kube-api-access-dz2c9\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5\" (UID: \"c8172c42-332c-40e4-9317-9014c5749253\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" Nov 03 14:23:25.334334 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.334312 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-041fbab7-4pp7d"] Nov 03 14:23:25.336845 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:23:25.336798 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod683f2838_8f53_4f9e_8bb5_a156ca3ff2d8.slice/crio-3fe1ae60e50dc0771d1ba20fa98239f5f703f7044de509b493cc8572d09a4600 WatchSource:0}: Error finding container 3fe1ae60e50dc0771d1ba20fa98239f5f703f7044de509b493cc8572d09a4600: Status 404 returned error can't find the container with id 3fe1ae60e50dc0771d1ba20fa98239f5f703f7044de509b493cc8572d09a4600 Nov 03 14:23:25.372060 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.372033 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-041fbab7-4pp7d" event={"ID":"683f2838-8f53-4f9e-8bb5-a156ca3ff2d8","Type":"ContainerStarted","Data":"3fe1ae60e50dc0771d1ba20fa98239f5f703f7044de509b493cc8572d09a4600"} Nov 03 14:23:25.402610 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.402580 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a88d21ce-086a-414a-911a-5561e2cb8023-istiod-ca-cert\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8\" (UID: \"a88d21ce-086a-414a-911a-5561e2cb8023\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" Nov 03 14:23:25.402757 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.402624 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a88d21ce-086a-414a-911a-5561e2cb8023-istio-token\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8\" (UID: \"a88d21ce-086a-414a-911a-5561e2cb8023\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" Nov 03 14:23:25.402757 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.402680 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a88d21ce-086a-414a-911a-5561e2cb8023-istio-data\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8\" (UID: \"a88d21ce-086a-414a-911a-5561e2cb8023\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" Nov 03 14:23:25.402757 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.402701 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a88d21ce-086a-414a-911a-5561e2cb8023-workload-certs\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8\" (UID: \"a88d21ce-086a-414a-911a-5561e2cb8023\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" Nov 03 14:23:25.402903 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.402851 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tms5j\" (UniqueName: \"kubernetes.io/projected/a88d21ce-086a-414a-911a-5561e2cb8023-kube-api-access-tms5j\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8\" (UID: \"a88d21ce-086a-414a-911a-5561e2cb8023\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" Nov 03 14:23:25.402965 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.402904 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a88d21ce-086a-414a-911a-5561e2cb8023-workload-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8\" (UID: \"a88d21ce-086a-414a-911a-5561e2cb8023\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" Nov 03 14:23:25.402965 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.402928 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a88d21ce-086a-414a-911a-5561e2cb8023-credential-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8\" (UID: \"a88d21ce-086a-414a-911a-5561e2cb8023\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" Nov 03 14:23:25.403073 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.403043 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a88d21ce-086a-414a-911a-5561e2cb8023-istio-podinfo\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8\" (UID: \"a88d21ce-086a-414a-911a-5561e2cb8023\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" Nov 03 14:23:25.403129 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.403070 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a88d21ce-086a-414a-911a-5561e2cb8023-istio-envoy\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8\" (UID: \"a88d21ce-086a-414a-911a-5561e2cb8023\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" Nov 03 14:23:25.403184 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.403130 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a88d21ce-086a-414a-911a-5561e2cb8023-istio-data\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8\" (UID: \"a88d21ce-086a-414a-911a-5561e2cb8023\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" Nov 03 14:23:25.403184 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.403156 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a88d21ce-086a-414a-911a-5561e2cb8023-workload-certs\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8\" (UID: \"a88d21ce-086a-414a-911a-5561e2cb8023\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" Nov 03 14:23:25.403364 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.403345 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a88d21ce-086a-414a-911a-5561e2cb8023-credential-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8\" (UID: \"a88d21ce-086a-414a-911a-5561e2cb8023\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" Nov 03 14:23:25.403436 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.403413 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a88d21ce-086a-414a-911a-5561e2cb8023-istiod-ca-cert\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8\" (UID: \"a88d21ce-086a-414a-911a-5561e2cb8023\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" Nov 03 14:23:25.403491 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.403439 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a88d21ce-086a-414a-911a-5561e2cb8023-workload-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8\" (UID: \"a88d21ce-086a-414a-911a-5561e2cb8023\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" Nov 03 14:23:25.405280 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.405251 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a88d21ce-086a-414a-911a-5561e2cb8023-istio-envoy\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8\" (UID: \"a88d21ce-086a-414a-911a-5561e2cb8023\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" Nov 03 14:23:25.405420 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.405397 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a88d21ce-086a-414a-911a-5561e2cb8023-istio-podinfo\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8\" (UID: \"a88d21ce-086a-414a-911a-5561e2cb8023\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" Nov 03 14:23:25.422670 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.422644 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a88d21ce-086a-414a-911a-5561e2cb8023-istio-token\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8\" (UID: \"a88d21ce-086a-414a-911a-5561e2cb8023\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" Nov 03 14:23:25.422766 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.422674 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tms5j\" (UniqueName: \"kubernetes.io/projected/a88d21ce-086a-414a-911a-5561e2cb8023-kube-api-access-tms5j\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8\" (UID: \"a88d21ce-086a-414a-911a-5561e2cb8023\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" Nov 03 14:23:25.430530 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.430509 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" Nov 03 14:23:25.584008 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.583921 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" Nov 03 14:23:25.749014 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.748973 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5"] Nov 03 14:23:25.753003 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:23:25.752969 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8172c42_332c_40e4_9317_9014c5749253.slice/crio-eca94d3ad163e97b83e466654963add16b12f04a0c1128130243cc9c9bd83f78 WatchSource:0}: Error finding container eca94d3ad163e97b83e466654963add16b12f04a0c1128130243cc9c9bd83f78: Status 404 returned error can't find the container with id eca94d3ad163e97b83e466654963add16b12f04a0c1128130243cc9c9bd83f78 Nov 03 14:23:25.756204 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.756168 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:23:25.756313 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.756248 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:23:25.756313 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.756297 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:23:25.971908 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.970949 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8"] Nov 03 14:23:25.993170 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.993116 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:23:25.993421 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.993397 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:23:25.993656 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:25.993625 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:23:26.380004 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:26.379966 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" event={"ID":"c8172c42-332c-40e4-9317-9014c5749253","Type":"ContainerStarted","Data":"a1519cd1e4d0fa11fa27ab35ab2f35597611cc362f25997eae15530a155c8208"} Nov 03 14:23:26.380192 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:26.380021 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" event={"ID":"c8172c42-332c-40e4-9317-9014c5749253","Type":"ContainerStarted","Data":"eca94d3ad163e97b83e466654963add16b12f04a0c1128130243cc9c9bd83f78"} Nov 03 14:23:26.381637 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:26.381603 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" event={"ID":"a88d21ce-086a-414a-911a-5561e2cb8023","Type":"ContainerStarted","Data":"ee16f02a8ea2bfaf37abb4dfb62eddb03226e4762ce4d44ab6356921a3528b84"} Nov 03 14:23:26.381770 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:26.381659 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" event={"ID":"a88d21ce-086a-414a-911a-5561e2cb8023","Type":"ContainerStarted","Data":"09d8fb217c248d63a80c981d291aceb2b5a68e20a47167d66006b61371516931"} Nov 03 14:23:26.383484 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:26.383458 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-041fbab7-4pp7d" event={"ID":"683f2838-8f53-4f9e-8bb5-a156ca3ff2d8","Type":"ContainerStarted","Data":"33675e8ffb2db0a802293f82bcf2fc5f086170de4e3d1c555f1b0217d1703352"} Nov 03 14:23:26.427678 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:26.427612 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" podStartSLOduration=1.4275908290000001 podStartE2EDuration="1.427590829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:23:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:23:26.421581793 +0000 UTC m=+740.310371377" watchObservedRunningTime="2026-11-03 14:23:26.427590829 +0000 UTC m=+740.316380412" Nov 03 14:23:26.433886 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:26.432773 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" Nov 03 14:23:26.433886 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:26.432905 5060 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 14:23:26.433886 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:26.432951 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" podUID="c8172c42-332c-40e4-9317-9014c5749253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 14:23:26.468932 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:26.468874 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" podStartSLOduration=1.46886076 podStartE2EDuration="1.46886076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:23:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:23:26.46556202 +0000 UTC m=+740.354351601" watchObservedRunningTime="2026-11-03 14:23:26.46886076 +0000 UTC m=+740.357650340" Nov 03 14:23:26.544240 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:26.542472 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-041fbab7-4pp7d" podStartSLOduration=2.542451427 podStartE2EDuration="2.542451427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:23:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:23:26.538215355 +0000 UTC m=+740.427004938" watchObservedRunningTime="2026-11-03 14:23:26.542451427 +0000 UTC m=+740.431241010" Nov 03 14:23:26.584069 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:26.584034 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" Nov 03 14:23:27.390948 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:27.390909 5060 generic.go:356] "Generic (PLEG): container finished" podID="683f2838-8f53-4f9e-8bb5-a156ca3ff2d8" containerID="33675e8ffb2db0a802293f82bcf2fc5f086170de4e3d1c555f1b0217d1703352" exitCode=0 Nov 03 14:23:27.391426 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:27.391011 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-041fbab7-4pp7d" event={"ID":"683f2838-8f53-4f9e-8bb5-a156ca3ff2d8","Type":"ContainerDied","Data":"33675e8ffb2db0a802293f82bcf2fc5f086170de4e3d1c555f1b0217d1703352"} Nov 03 14:23:27.431856 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:27.431281 5060 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 14:23:27.431856 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:27.431360 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" podUID="c8172c42-332c-40e4-9317-9014c5749253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 14:23:27.586095 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:27.585953 5060 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 14:23:27.586095 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:27.586016 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" podUID="a88d21ce-086a-414a-911a-5561e2cb8023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 14:23:28.431671 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:28.431228 5060 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 14:23:28.431671 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:28.431292 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" podUID="c8172c42-332c-40e4-9317-9014c5749253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 14:23:28.479696 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:28.479651 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98" Nov 03 14:23:28.495725 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:28.495700 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98" Nov 03 14:23:28.585203 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:28.585169 5060 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 14:23:28.585460 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:28.585424 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" podUID="a88d21ce-086a-414a-911a-5561e2cb8023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 14:23:28.771588 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:28.771561 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-041fbab7-4pp7d" Nov 03 14:23:28.777250 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:28.777226 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/683f2838-8f53-4f9e-8bb5-a156ca3ff2d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/683f2838-8f53-4f9e-8bb5-a156ca3ff2d8-maas-api-xks-ca\") pod \"683f2838-8f53-4f9e-8bb5-a156ca3ff2d8\" (UID: \"683f2838-8f53-4f9e-8bb5-a156ca3ff2d8\") " Nov 03 14:23:28.777411 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:28.777395 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/683f2838-8f53-4f9e-8bb5-a156ca3ff2d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/683f2838-8f53-4f9e-8bb5-a156ca3ff2d8-maas-api-openshift-ca\") pod \"683f2838-8f53-4f9e-8bb5-a156ca3ff2d8\" (UID: \"683f2838-8f53-4f9e-8bb5-a156ca3ff2d8\") " Nov 03 14:23:28.777450 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:28.777434 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/683f2838-8f53-4f9e-8bb5-a156ca3ff2d8-kube-api-access-rcmd7\" (UniqueName: \"kubernetes.io/projected/683f2838-8f53-4f9e-8bb5-a156ca3ff2d8-kube-api-access-rcmd7\") pod \"683f2838-8f53-4f9e-8bb5-a156ca3ff2d8\" (UID: \"683f2838-8f53-4f9e-8bb5-a156ca3ff2d8\") " Nov 03 14:23:28.777757 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:28.777732 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/683f2838-8f53-4f9e-8bb5-a156ca3ff2d8-maas-api-openshift-ca" pod "683f2838-8f53-4f9e-8bb5-a156ca3ff2d8" (UID: "683f2838-8f53-4f9e-8bb5-a156ca3ff2d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:23:28.783210 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:28.783145 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/683f2838-8f53-4f9e-8bb5-a156ca3ff2d8-maas-api-xks-ca" pod "683f2838-8f53-4f9e-8bb5-a156ca3ff2d8" (UID: "683f2838-8f53-4f9e-8bb5-a156ca3ff2d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:23:28.783210 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:28.783188 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/683f2838-8f53-4f9e-8bb5-a156ca3ff2d8-kube-api-access-rcmd7" pod "683f2838-8f53-4f9e-8bb5-a156ca3ff2d8" (UID: "683f2838-8f53-4f9e-8bb5-a156ca3ff2d8"). InnerVolumeSpecName "kube-api-access-rcmd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:23:28.878949 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:28.878671 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/683f2838-8f53-4f9e-8bb5-a156ca3ff2d8-maas-api-openshift-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:23:28.878949 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:28.878709 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcmd7\" (UniqueName: \"kubernetes.io/projected/683f2838-8f53-4f9e-8bb5-a156ca3ff2d8-kube-api-access-rcmd7\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:23:28.878949 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:28.878725 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/683f2838-8f53-4f9e-8bb5-a156ca3ff2d8-maas-api-xks-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:23:29.418512 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:29.418482 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-041fbab7-4pp7d" Nov 03 14:23:29.423504 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:29.423472 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-041fbab7-4pp7d" event={"ID":"683f2838-8f53-4f9e-8bb5-a156ca3ff2d8","Type":"ContainerDied","Data":"3fe1ae60e50dc0771d1ba20fa98239f5f703f7044de509b493cc8572d09a4600"} Nov 03 14:23:29.423693 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:29.423676 5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fe1ae60e50dc0771d1ba20fa98239f5f703f7044de509b493cc8572d09a4600" Nov 03 14:23:29.432307 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:29.432274 5060 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 14:23:29.432814 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:29.432788 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" podUID="c8172c42-332c-40e4-9317-9014c5749253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 14:23:29.470858 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:29.468759 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98" Nov 03 14:23:29.585306 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:29.585272 5060 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 14:23:29.585499 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:29.585330 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" podUID="a88d21ce-086a-414a-911a-5561e2cb8023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 14:23:30.441859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:30.441375 5060 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 14:23:30.442349 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:30.441871 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" podUID="c8172c42-332c-40e4-9317-9014c5749253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 14:23:30.584762 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:30.584728 5060 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 14:23:30.584957 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:30.584780 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" podUID="a88d21ce-086a-414a-911a-5561e2cb8023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 14:23:31.432860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:31.432368 5060 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 14:23:31.432860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:31.432442 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" podUID="c8172c42-332c-40e4-9317-9014c5749253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 14:23:31.584785 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:31.584749 5060 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 14:23:31.585200 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:31.584810 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" podUID="a88d21ce-086a-414a-911a-5561e2cb8023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 14:23:32.431190 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:32.431155 5060 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 14:23:32.431439 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:32.431409 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" podUID="c8172c42-332c-40e4-9317-9014c5749253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 14:23:32.584761 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:32.584718 5060 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 14:23:32.584946 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:32.584789 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" podUID="a88d21ce-086a-414a-911a-5561e2cb8023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 14:23:33.771406 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:33.771369 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" Nov 03 14:23:33.771897 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:33.771777 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" Nov 03 14:23:33.772691 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:33.772668 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" Nov 03 14:23:34.144084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:34.143990 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" Nov 03 14:23:34.145120 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:34.145094 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" Nov 03 14:23:34.148069 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:34.148053 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" Nov 03 14:23:39.440853 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:23:39.440799 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:23:40.335538 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:40.335505 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v"] Nov 03 14:23:40.336275 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:40.336250 5060 cpu_manager.go:414] "containerMap: removing container" podUID="683f2838-8f53-4f9e-8bb5-a156ca3ff2d8" containerName="revoke-keys" Nov 03 14:23:40.336275 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:40.336271 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="683f2838-8f53-4f9e-8bb5-a156ca3ff2d8" containerName="revoke-keys" Nov 03 14:23:40.336468 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:40.336403 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="683f2838-8f53-4f9e-8bb5-a156ca3ff2d8" containerName="revoke-keys" Nov 03 14:23:40.343363 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:40.343332 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" Nov 03 14:23:40.372700 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:40.372430 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v"] Nov 03 14:23:40.453804 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:40.453768 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6695b0e-c9b6-4119-a0b2-508b212a3ece-workload-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v\" (UID: \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" Nov 03 14:23:40.454226 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:40.453823 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6695b0e-c9b6-4119-a0b2-508b212a3ece-workload-certs\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v\" (UID: \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" Nov 03 14:23:40.454226 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:40.453933 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6695b0e-c9b6-4119-a0b2-508b212a3ece-istio-token\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v\" (UID: \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" Nov 03 14:23:40.454226 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:40.453974 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6695b0e-c9b6-4119-a0b2-508b212a3ece-istiod-ca-cert\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v\" (UID: \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" Nov 03 14:23:40.454226 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:40.454001 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6695b0e-c9b6-4119-a0b2-508b212a3ece-istio-podinfo\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v\" (UID: \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" Nov 03 14:23:40.454226 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:40.454027 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4hq6\" (UniqueName: \"kubernetes.io/projected/f6695b0e-c9b6-4119-a0b2-508b212a3ece-kube-api-access-v4hq6\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v\" (UID: \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" Nov 03 14:23:40.454226 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:40.454120 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6695b0e-c9b6-4119-a0b2-508b212a3ece-istio-envoy\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v\" (UID: \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" Nov 03 14:23:40.454226 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:40.454189 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6695b0e-c9b6-4119-a0b2-508b212a3ece-istio-data\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v\" (UID: \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" Nov 03 14:23:40.454226 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:40.454211 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6695b0e-c9b6-4119-a0b2-508b212a3ece-credential-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v\" (UID: \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" Nov 03 14:23:40.555562 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:40.555518 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6695b0e-c9b6-4119-a0b2-508b212a3ece-istiod-ca-cert\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v\" (UID: \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" Nov 03 14:23:40.555740 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:40.555583 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6695b0e-c9b6-4119-a0b2-508b212a3ece-istio-podinfo\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v\" (UID: \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" Nov 03 14:23:40.555740 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:40.555625 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4hq6\" (UniqueName: \"kubernetes.io/projected/f6695b0e-c9b6-4119-a0b2-508b212a3ece-kube-api-access-v4hq6\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v\" (UID: \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" Nov 03 14:23:40.555740 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:40.555700 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6695b0e-c9b6-4119-a0b2-508b212a3ece-istio-envoy\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v\" (UID: \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" Nov 03 14:23:40.555973 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:40.555767 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6695b0e-c9b6-4119-a0b2-508b212a3ece-istio-data\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v\" (UID: \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" Nov 03 14:23:40.555973 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:40.555796 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6695b0e-c9b6-4119-a0b2-508b212a3ece-credential-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v\" (UID: \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" Nov 03 14:23:40.555973 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:40.555878 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6695b0e-c9b6-4119-a0b2-508b212a3ece-workload-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v\" (UID: \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" Nov 03 14:23:40.555973 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:40.555926 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6695b0e-c9b6-4119-a0b2-508b212a3ece-workload-certs\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v\" (UID: \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" Nov 03 14:23:40.555973 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:40.555972 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6695b0e-c9b6-4119-a0b2-508b212a3ece-istio-token\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v\" (UID: \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" Nov 03 14:23:40.556224 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:40.556185 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6695b0e-c9b6-4119-a0b2-508b212a3ece-istio-data\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v\" (UID: \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" Nov 03 14:23:40.556357 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:40.556332 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6695b0e-c9b6-4119-a0b2-508b212a3ece-credential-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v\" (UID: \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" Nov 03 14:23:40.556418 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:40.556384 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6695b0e-c9b6-4119-a0b2-508b212a3ece-workload-certs\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v\" (UID: \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" Nov 03 14:23:40.556418 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:40.556383 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6695b0e-c9b6-4119-a0b2-508b212a3ece-workload-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v\" (UID: \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" Nov 03 14:23:40.556515 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:40.556449 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6695b0e-c9b6-4119-a0b2-508b212a3ece-istiod-ca-cert\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v\" (UID: \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" Nov 03 14:23:40.560865 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:40.558170 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6695b0e-c9b6-4119-a0b2-508b212a3ece-istio-envoy\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v\" (UID: \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" Nov 03 14:23:40.560865 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:40.558681 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6695b0e-c9b6-4119-a0b2-508b212a3ece-istio-podinfo\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v\" (UID: \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" Nov 03 14:23:40.577157 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:40.577129 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4hq6\" (UniqueName: \"kubernetes.io/projected/f6695b0e-c9b6-4119-a0b2-508b212a3ece-kube-api-access-v4hq6\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v\" (UID: \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" Nov 03 14:23:40.578022 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:40.578000 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6695b0e-c9b6-4119-a0b2-508b212a3ece-istio-token\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v\" (UID: \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" Nov 03 14:23:40.657511 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:40.657416 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" Nov 03 14:23:41.001301 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:41.001272 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v"] Nov 03 14:23:41.003169 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:23:41.003128 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6695b0e_c9b6_4119_a0b2_508b212a3ece.slice/crio-d2e3f15e556a5e9a39c5116cf348bb3e4079f3d353d4b4584a127dce278f5406 WatchSource:0}: Error finding container d2e3f15e556a5e9a39c5116cf348bb3e4079f3d353d4b4584a127dce278f5406: Status 404 returned error can't find the container with id d2e3f15e556a5e9a39c5116cf348bb3e4079f3d353d4b4584a127dce278f5406 Nov 03 14:23:41.005794 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:41.005762 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:23:41.005926 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:41.005822 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:23:41.005926 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:41.005885 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:23:41.571502 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:41.560269 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" event={"ID":"f6695b0e-c9b6-4119-a0b2-508b212a3ece","Type":"ContainerStarted","Data":"b6f1d2902246456c639f4b454edd5fbb2ddab0a2d1c3474d794c7fdcb26b6673"} Nov 03 14:23:41.571502 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:41.560343 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" event={"ID":"f6695b0e-c9b6-4119-a0b2-508b212a3ece","Type":"ContainerStarted","Data":"d2e3f15e556a5e9a39c5116cf348bb3e4079f3d353d4b4584a127dce278f5406"} Nov 03 14:23:41.659235 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:41.659167 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" podStartSLOduration=1.65915064 podStartE2EDuration="1.65915064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:23:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:23:41.647237266 +0000 UTC m=+755.536026849" watchObservedRunningTime="2026-11-03 14:23:41.65915064 +0000 UTC m=+755.547940220" Nov 03 14:23:41.659427 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:41.659334 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" Nov 03 14:23:42.675859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:42.667793 5060 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 14:23:42.676373 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:42.675999 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" podUID="f6695b0e-c9b6-4119-a0b2-508b212a3ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 14:23:43.662857 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:43.662554 5060 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 14:23:43.662857 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:43.662612 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" podUID="f6695b0e-c9b6-4119-a0b2-508b212a3ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 14:23:44.660275 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:44.659981 5060 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 14:23:44.660275 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:44.660042 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" podUID="f6695b0e-c9b6-4119-a0b2-508b212a3ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 14:23:45.660275 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:45.660235 5060 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 14:23:45.660586 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:45.660557 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" podUID="f6695b0e-c9b6-4119-a0b2-508b212a3ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 14:23:46.658514 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:46.658470 5060 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 14:23:46.658817 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:46.658535 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" podUID="f6695b0e-c9b6-4119-a0b2-508b212a3ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 14:23:47.658379 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:47.658336 5060 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 14:23:47.658749 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:47.658401 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" podUID="f6695b0e-c9b6-4119-a0b2-508b212a3ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 14:23:48.732151 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:48.732093 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" Nov 03 14:23:48.733452 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:48.733426 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" Nov 03 14:23:48.734653 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:48.734629 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" Nov 03 14:23:48.834035 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:48.833989 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-156ae1/distinct-2-w3-156ae1-kserve-5988894d76-vsdlm"] Nov 03 14:23:48.842137 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:48.842108 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-156ae1/distinct-2-w3-156ae1-kserve-5988894d76-vsdlm" Nov 03 14:23:48.859322 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:48.859287 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-156ae1\"/\"distinct-2-w3-156ae1-kserve-self-signed-certs\"" Nov 03 14:23:48.871559 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:48.870163 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59dcdef3-f88e-4463-a0dd-1917e915a516-tmp-dir\") pod \"distinct-2-w3-156ae1-kserve-5988894d76-vsdlm\" (UID: \"59dcdef3-f88e-4463-a0dd-1917e915a516\") " pod="e2e-models-e2e-worker-w3-156ae1/distinct-2-w3-156ae1-kserve-5988894d76-vsdlm" Nov 03 14:23:48.871559 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:48.870210 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59dcdef3-f88e-4463-a0dd-1917e915a516-model-cache\") pod \"distinct-2-w3-156ae1-kserve-5988894d76-vsdlm\" (UID: \"59dcdef3-f88e-4463-a0dd-1917e915a516\") " pod="e2e-models-e2e-worker-w3-156ae1/distinct-2-w3-156ae1-kserve-5988894d76-vsdlm" Nov 03 14:23:48.872212 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:48.871924 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59dcdef3-f88e-4463-a0dd-1917e915a516-home\") pod \"distinct-2-w3-156ae1-kserve-5988894d76-vsdlm\" (UID: \"59dcdef3-f88e-4463-a0dd-1917e915a516\") " pod="e2e-models-e2e-worker-w3-156ae1/distinct-2-w3-156ae1-kserve-5988894d76-vsdlm" Nov 03 14:23:48.872212 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:48.871973 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggq2x\" (UniqueName: \"kubernetes.io/projected/59dcdef3-f88e-4463-a0dd-1917e915a516-kube-api-access-ggq2x\") pod \"distinct-2-w3-156ae1-kserve-5988894d76-vsdlm\" (UID: \"59dcdef3-f88e-4463-a0dd-1917e915a516\") " pod="e2e-models-e2e-worker-w3-156ae1/distinct-2-w3-156ae1-kserve-5988894d76-vsdlm" Nov 03 14:23:48.872212 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:48.872003 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59dcdef3-f88e-4463-a0dd-1917e915a516-tls-certs\") pod \"distinct-2-w3-156ae1-kserve-5988894d76-vsdlm\" (UID: \"59dcdef3-f88e-4463-a0dd-1917e915a516\") " pod="e2e-models-e2e-worker-w3-156ae1/distinct-2-w3-156ae1-kserve-5988894d76-vsdlm" Nov 03 14:23:48.872212 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:48.872057 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59dcdef3-f88e-4463-a0dd-1917e915a516-dshm\") pod \"distinct-2-w3-156ae1-kserve-5988894d76-vsdlm\" (UID: \"59dcdef3-f88e-4463-a0dd-1917e915a516\") " pod="e2e-models-e2e-worker-w3-156ae1/distinct-2-w3-156ae1-kserve-5988894d76-vsdlm" Nov 03 14:23:48.877567 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:48.877535 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-156ae1/distinct-2-w3-156ae1-kserve-5988894d76-vsdlm"] Nov 03 14:23:48.972613 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:48.972580 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59dcdef3-f88e-4463-a0dd-1917e915a516-tmp-dir\") pod \"distinct-2-w3-156ae1-kserve-5988894d76-vsdlm\" (UID: \"59dcdef3-f88e-4463-a0dd-1917e915a516\") " pod="e2e-models-e2e-worker-w3-156ae1/distinct-2-w3-156ae1-kserve-5988894d76-vsdlm" Nov 03 14:23:48.972786 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:48.972620 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59dcdef3-f88e-4463-a0dd-1917e915a516-model-cache\") pod \"distinct-2-w3-156ae1-kserve-5988894d76-vsdlm\" (UID: \"59dcdef3-f88e-4463-a0dd-1917e915a516\") " pod="e2e-models-e2e-worker-w3-156ae1/distinct-2-w3-156ae1-kserve-5988894d76-vsdlm" Nov 03 14:23:48.972786 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:48.972735 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59dcdef3-f88e-4463-a0dd-1917e915a516-home\") pod \"distinct-2-w3-156ae1-kserve-5988894d76-vsdlm\" (UID: \"59dcdef3-f88e-4463-a0dd-1917e915a516\") " pod="e2e-models-e2e-worker-w3-156ae1/distinct-2-w3-156ae1-kserve-5988894d76-vsdlm" Nov 03 14:23:48.972786 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:48.972769 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggq2x\" (UniqueName: \"kubernetes.io/projected/59dcdef3-f88e-4463-a0dd-1917e915a516-kube-api-access-ggq2x\") pod \"distinct-2-w3-156ae1-kserve-5988894d76-vsdlm\" (UID: \"59dcdef3-f88e-4463-a0dd-1917e915a516\") " pod="e2e-models-e2e-worker-w3-156ae1/distinct-2-w3-156ae1-kserve-5988894d76-vsdlm" Nov 03 14:23:48.972909 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:48.972809 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59dcdef3-f88e-4463-a0dd-1917e915a516-tls-certs\") pod \"distinct-2-w3-156ae1-kserve-5988894d76-vsdlm\" (UID: \"59dcdef3-f88e-4463-a0dd-1917e915a516\") " pod="e2e-models-e2e-worker-w3-156ae1/distinct-2-w3-156ae1-kserve-5988894d76-vsdlm" Nov 03 14:23:48.972909 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:48.972902 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59dcdef3-f88e-4463-a0dd-1917e915a516-dshm\") pod \"distinct-2-w3-156ae1-kserve-5988894d76-vsdlm\" (UID: \"59dcdef3-f88e-4463-a0dd-1917e915a516\") " pod="e2e-models-e2e-worker-w3-156ae1/distinct-2-w3-156ae1-kserve-5988894d76-vsdlm" Nov 03 14:23:48.973113 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:48.973076 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59dcdef3-f88e-4463-a0dd-1917e915a516-tmp-dir\") pod \"distinct-2-w3-156ae1-kserve-5988894d76-vsdlm\" (UID: \"59dcdef3-f88e-4463-a0dd-1917e915a516\") " pod="e2e-models-e2e-worker-w3-156ae1/distinct-2-w3-156ae1-kserve-5988894d76-vsdlm" Nov 03 14:23:48.973669 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:48.973356 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59dcdef3-f88e-4463-a0dd-1917e915a516-home\") pod \"distinct-2-w3-156ae1-kserve-5988894d76-vsdlm\" (UID: \"59dcdef3-f88e-4463-a0dd-1917e915a516\") " pod="e2e-models-e2e-worker-w3-156ae1/distinct-2-w3-156ae1-kserve-5988894d76-vsdlm" Nov 03 14:23:48.973669 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:48.973609 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59dcdef3-f88e-4463-a0dd-1917e915a516-model-cache\") pod \"distinct-2-w3-156ae1-kserve-5988894d76-vsdlm\" (UID: \"59dcdef3-f88e-4463-a0dd-1917e915a516\") " pod="e2e-models-e2e-worker-w3-156ae1/distinct-2-w3-156ae1-kserve-5988894d76-vsdlm" Nov 03 14:23:48.975499 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:48.975478 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59dcdef3-f88e-4463-a0dd-1917e915a516-dshm\") pod \"distinct-2-w3-156ae1-kserve-5988894d76-vsdlm\" (UID: \"59dcdef3-f88e-4463-a0dd-1917e915a516\") " pod="e2e-models-e2e-worker-w3-156ae1/distinct-2-w3-156ae1-kserve-5988894d76-vsdlm" Nov 03 14:23:48.976452 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:48.976432 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59dcdef3-f88e-4463-a0dd-1917e915a516-tls-certs\") pod \"distinct-2-w3-156ae1-kserve-5988894d76-vsdlm\" (UID: \"59dcdef3-f88e-4463-a0dd-1917e915a516\") " pod="e2e-models-e2e-worker-w3-156ae1/distinct-2-w3-156ae1-kserve-5988894d76-vsdlm" Nov 03 14:23:48.994063 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:48.994035 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggq2x\" (UniqueName: \"kubernetes.io/projected/59dcdef3-f88e-4463-a0dd-1917e915a516-kube-api-access-ggq2x\") pod \"distinct-2-w3-156ae1-kserve-5988894d76-vsdlm\" (UID: \"59dcdef3-f88e-4463-a0dd-1917e915a516\") " pod="e2e-models-e2e-worker-w3-156ae1/distinct-2-w3-156ae1-kserve-5988894d76-vsdlm" Nov 03 14:23:49.182702 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:49.182665 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-156ae1/distinct-2-w3-156ae1-kserve-5988894d76-vsdlm" Nov 03 14:23:49.534623 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:49.534593 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-156ae1/distinct-2-w3-156ae1-kserve-5988894d76-vsdlm"] Nov 03 14:23:49.537303 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:23:49.537271 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59dcdef3_f88e_4463_a0dd_1917e915a516.slice/crio-70479f9753393fdc2f096e381f9f395a6df94e8c97cc8499ec88263d059ef830 WatchSource:0}: Error finding container 70479f9753393fdc2f096e381f9f395a6df94e8c97cc8499ec88263d059ef830: Status 404 returned error can't find the container with id 70479f9753393fdc2f096e381f9f395a6df94e8c97cc8499ec88263d059ef830 Nov 03 14:23:49.655814 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:49.655773 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-156ae1/distinct-2-w3-156ae1-kserve-5988894d76-vsdlm" event={"ID":"59dcdef3-f88e-4463-a0dd-1917e915a516","Type":"ContainerStarted","Data":"b6f58148cd1a1713b2b050d10667987ec0f3bb884f5a9a08eb38c9dd50c90ad9"} Nov 03 14:23:49.656014 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:49.655827 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-156ae1/distinct-2-w3-156ae1-kserve-5988894d76-vsdlm" event={"ID":"59dcdef3-f88e-4463-a0dd-1917e915a516","Type":"ContainerStarted","Data":"70479f9753393fdc2f096e381f9f395a6df94e8c97cc8499ec88263d059ef830"} Nov 03 14:23:49.702343 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:49.702293 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-156ae1/distinct-2-w3-156ae1-kserve-5988894d76-vsdlm" podStartSLOduration=1.702277562 podStartE2EDuration="1.702277562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:23:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:23:49.699425074 +0000 UTC m=+763.588214655" watchObservedRunningTime="2026-11-03 14:23:49.702277562 +0000 UTC m=+763.591067143" Nov 03 14:23:50.183512 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:50.183466 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-156ae1/distinct-2-w3-156ae1-kserve-5988894d76-vsdlm" Nov 03 14:23:51.669759 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:51.669727 5060 generic.go:356] "Generic (PLEG): container finished" podID="711e30e5-2c44-4295-974c-80a266623ae1" containerID="0ba1c7f2a6096b1f7b4f7d4f5e68ed118f027a5f9427f6aa40d4ed459eba6fab" exitCode=0 Nov 03 14:23:51.670208 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:51.669796 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" event={"ID":"711e30e5-2c44-4295-974c-80a266623ae1","Type":"ContainerDied","Data":"0ba1c7f2a6096b1f7b4f7d4f5e68ed118f027a5f9427f6aa40d4ed459eba6fab"} Nov 03 14:23:51.670283 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:51.670269 5060 scope.go:122] "RemoveContainer" containerID="0ba1c7f2a6096b1f7b4f7d4f5e68ed118f027a5f9427f6aa40d4ed459eba6fab" Nov 03 14:23:51.671548 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:51.671525 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:23:51.671608 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:51.671596 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:23:51.671644 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:51.671627 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:23:52.678315 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:52.678276 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" event={"ID":"711e30e5-2c44-4295-974c-80a266623ae1","Type":"ContainerStarted","Data":"e74b2bfd81eb5d47beeb2ee25d5bd0af3f6d161ad1c4f4145f9f17dcd8e75039"} Nov 03 14:23:53.580555 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:53.580517 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" Nov 03 14:23:53.580734 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:53.580700 5060 patch_prober.go:29] interesting pod/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 03 14:23:53.580783 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:53.580747 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" podUID="711e30e5-2c44-4295-974c-80a266623ae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 03 14:23:54.582676 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:54.582635 5060 patch_prober.go:29] interesting pod/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 03 14:23:54.583301 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:54.583267 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" podUID="711e30e5-2c44-4295-974c-80a266623ae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 03 14:23:55.192249 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:55.192197 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw"] Nov 03 14:23:55.210586 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:55.210554 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" Nov 03 14:23:55.306431 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:55.306400 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw"] Nov 03 14:23:55.336558 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:55.336527 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e759e5-openshift-default-dockercfg-ldb9g\"" Nov 03 14:23:55.418923 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:55.398316 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c60d54c8-1f41-4f9c-9401-63a07e12f664-credential-socket\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw\" (UID: \"c60d54c8-1f41-4f9c-9401-63a07e12f664\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" Nov 03 14:23:55.418923 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:55.398372 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c60d54c8-1f41-4f9c-9401-63a07e12f664-istio-data\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw\" (UID: \"c60d54c8-1f41-4f9c-9401-63a07e12f664\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" Nov 03 14:23:55.418923 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:55.398422 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c60d54c8-1f41-4f9c-9401-63a07e12f664-istio-podinfo\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw\" (UID: \"c60d54c8-1f41-4f9c-9401-63a07e12f664\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" Nov 03 14:23:55.418923 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:55.398446 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c60d54c8-1f41-4f9c-9401-63a07e12f664-istio-token\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw\" (UID: \"c60d54c8-1f41-4f9c-9401-63a07e12f664\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" Nov 03 14:23:55.418923 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:55.398498 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c60d54c8-1f41-4f9c-9401-63a07e12f664-istiod-ca-cert\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw\" (UID: \"c60d54c8-1f41-4f9c-9401-63a07e12f664\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" Nov 03 14:23:55.418923 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:55.398529 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c60d54c8-1f41-4f9c-9401-63a07e12f664-workload-socket\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw\" (UID: \"c60d54c8-1f41-4f9c-9401-63a07e12f664\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" Nov 03 14:23:55.418923 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:55.398556 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c60d54c8-1f41-4f9c-9401-63a07e12f664-workload-certs\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw\" (UID: \"c60d54c8-1f41-4f9c-9401-63a07e12f664\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" Nov 03 14:23:55.418923 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:55.398625 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsppv\" (UniqueName: \"kubernetes.io/projected/c60d54c8-1f41-4f9c-9401-63a07e12f664-kube-api-access-wsppv\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw\" (UID: \"c60d54c8-1f41-4f9c-9401-63a07e12f664\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" Nov 03 14:23:55.418923 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:55.398657 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c60d54c8-1f41-4f9c-9401-63a07e12f664-istio-envoy\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw\" (UID: \"c60d54c8-1f41-4f9c-9401-63a07e12f664\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" Nov 03 14:23:55.508649 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:55.507634 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c60d54c8-1f41-4f9c-9401-63a07e12f664-credential-socket\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw\" (UID: \"c60d54c8-1f41-4f9c-9401-63a07e12f664\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" Nov 03 14:23:55.508649 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:55.507690 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c60d54c8-1f41-4f9c-9401-63a07e12f664-istio-data\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw\" (UID: \"c60d54c8-1f41-4f9c-9401-63a07e12f664\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" Nov 03 14:23:55.508649 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:55.507747 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c60d54c8-1f41-4f9c-9401-63a07e12f664-istio-podinfo\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw\" (UID: \"c60d54c8-1f41-4f9c-9401-63a07e12f664\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" Nov 03 14:23:55.508649 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:55.507771 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c60d54c8-1f41-4f9c-9401-63a07e12f664-istio-token\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw\" (UID: \"c60d54c8-1f41-4f9c-9401-63a07e12f664\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" Nov 03 14:23:55.508649 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:55.507826 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c60d54c8-1f41-4f9c-9401-63a07e12f664-istiod-ca-cert\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw\" (UID: \"c60d54c8-1f41-4f9c-9401-63a07e12f664\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" Nov 03 14:23:55.508649 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:55.507893 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c60d54c8-1f41-4f9c-9401-63a07e12f664-workload-socket\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw\" (UID: \"c60d54c8-1f41-4f9c-9401-63a07e12f664\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" Nov 03 14:23:55.508649 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:55.507923 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c60d54c8-1f41-4f9c-9401-63a07e12f664-workload-certs\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw\" (UID: \"c60d54c8-1f41-4f9c-9401-63a07e12f664\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" Nov 03 14:23:55.508649 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:55.507999 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsppv\" (UniqueName: \"kubernetes.io/projected/c60d54c8-1f41-4f9c-9401-63a07e12f664-kube-api-access-wsppv\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw\" (UID: \"c60d54c8-1f41-4f9c-9401-63a07e12f664\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" Nov 03 14:23:55.508649 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:55.508035 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c60d54c8-1f41-4f9c-9401-63a07e12f664-istio-envoy\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw\" (UID: \"c60d54c8-1f41-4f9c-9401-63a07e12f664\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" Nov 03 14:23:55.521857 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:55.521274 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c60d54c8-1f41-4f9c-9401-63a07e12f664-istio-data\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw\" (UID: \"c60d54c8-1f41-4f9c-9401-63a07e12f664\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" Nov 03 14:23:55.522650 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:55.522620 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c60d54c8-1f41-4f9c-9401-63a07e12f664-workload-socket\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw\" (UID: \"c60d54c8-1f41-4f9c-9401-63a07e12f664\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" Nov 03 14:23:55.524691 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:55.524471 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c60d54c8-1f41-4f9c-9401-63a07e12f664-workload-certs\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw\" (UID: \"c60d54c8-1f41-4f9c-9401-63a07e12f664\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" Nov 03 14:23:55.539603 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:55.539572 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c60d54c8-1f41-4f9c-9401-63a07e12f664-istiod-ca-cert\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw\" (UID: \"c60d54c8-1f41-4f9c-9401-63a07e12f664\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" Nov 03 14:23:55.560119 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:55.560033 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c60d54c8-1f41-4f9c-9401-63a07e12f664-credential-socket\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw\" (UID: \"c60d54c8-1f41-4f9c-9401-63a07e12f664\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" Nov 03 14:23:55.589148 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:55.588633 5060 patch_prober.go:29] interesting pod/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 03 14:23:55.589148 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:55.588711 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" podUID="711e30e5-2c44-4295-974c-80a266623ae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 03 14:23:55.617859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:55.590153 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c60d54c8-1f41-4f9c-9401-63a07e12f664-istio-envoy\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw\" (UID: \"c60d54c8-1f41-4f9c-9401-63a07e12f664\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" Nov 03 14:23:55.637300 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:55.619185 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c60d54c8-1f41-4f9c-9401-63a07e12f664-istio-token\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw\" (UID: \"c60d54c8-1f41-4f9c-9401-63a07e12f664\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" Nov 03 14:23:55.647942 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:55.638749 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c60d54c8-1f41-4f9c-9401-63a07e12f664-istio-podinfo\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw\" (UID: \"c60d54c8-1f41-4f9c-9401-63a07e12f664\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" Nov 03 14:23:55.647942 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:55.639393 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsppv\" (UniqueName: \"kubernetes.io/projected/c60d54c8-1f41-4f9c-9401-63a07e12f664-kube-api-access-wsppv\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw\" (UID: \"c60d54c8-1f41-4f9c-9401-63a07e12f664\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" Nov 03 14:23:55.827190 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:55.827043 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" Nov 03 14:23:56.680410 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:56.680327 5060 patch_prober.go:29] interesting pod/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 03 14:23:56.693558 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:56.688459 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" podUID="711e30e5-2c44-4295-974c-80a266623ae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 03 14:23:56.879732 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:56.879694 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw"] Nov 03 14:23:56.996778 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:56.996382 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:23:56.997263 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:56.997223 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:23:57.013897 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:56.997431 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:23:57.603339 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:57.594885 5060 patch_prober.go:29] interesting pod/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 03 14:23:57.603339 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:57.594952 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" podUID="711e30e5-2c44-4295-974c-80a266623ae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 03 14:23:57.868914 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:57.868357 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" event={"ID":"c60d54c8-1f41-4f9c-9401-63a07e12f664","Type":"ContainerStarted","Data":"1bd3a1b6dec9eb61ece036757e428efd0cfc141931c5ae8d2595dc0166c1c964"} Nov 03 14:23:57.868914 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:57.868449 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" event={"ID":"c60d54c8-1f41-4f9c-9401-63a07e12f664","Type":"ContainerStarted","Data":"f209e8f2594b60380867e8015eb1d40453d4a78ff19321e13ac60d9224ad3e36"} Nov 03 14:23:57.949885 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:57.947899 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" podStartSLOduration=2.947879676 podStartE2EDuration="2.947879676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:23:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:23:57.945690824 +0000 UTC m=+771.834480409" watchObservedRunningTime="2026-11-03 14:23:57.947879676 +0000 UTC m=+771.836669257" Nov 03 14:23:58.580056 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:58.580022 5060 patch_prober.go:29] interesting pod/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 03 14:23:58.580205 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:58.580076 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" podUID="711e30e5-2c44-4295-974c-80a266623ae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 03 14:23:58.828192 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:58.828156 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" Nov 03 14:23:58.833796 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:58.833765 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" Nov 03 14:23:58.877812 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:58.877779 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" Nov 03 14:23:58.879191 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:58.879124 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" Nov 03 14:23:59.183625 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:59.183552 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-156ae1/distinct-2-w3-156ae1-kserve-5988894d76-vsdlm" Nov 03 14:23:59.195776 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:59.195749 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-156ae1/distinct-2-w3-156ae1-kserve-5988894d76-vsdlm" Nov 03 14:23:59.584950 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:59.584917 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" Nov 03 14:23:59.585340 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:59.585317 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" Nov 03 14:23:59.585918 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:59.585901 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" Nov 03 14:23:59.893817 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:23:59.893739 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-156ae1/distinct-2-w3-156ae1-kserve-5988894d76-vsdlm" Nov 03 14:24:09.440467 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:24:09.440431 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:24:10.288474 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:10.288439 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg"] Nov 03 14:24:10.294060 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:10.294032 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" Nov 03 14:24:10.318081 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:10.318055 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg"] Nov 03 14:24:10.408221 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:10.408185 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f859590f-c6cb-4b6c-bfd2-ebde80002e81-istio-token\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg\" (UID: \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" Nov 03 14:24:10.408411 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:10.408253 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f859590f-c6cb-4b6c-bfd2-ebde80002e81-istio-envoy\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg\" (UID: \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" Nov 03 14:24:10.408411 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:10.408339 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f859590f-c6cb-4b6c-bfd2-ebde80002e81-workload-certs\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg\" (UID: \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" Nov 03 14:24:10.408411 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:10.408365 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f859590f-c6cb-4b6c-bfd2-ebde80002e81-istio-podinfo\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg\" (UID: \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" Nov 03 14:24:10.408556 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:10.408512 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w2v7\" (UniqueName: \"kubernetes.io/projected/f859590f-c6cb-4b6c-bfd2-ebde80002e81-kube-api-access-8w2v7\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg\" (UID: \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" Nov 03 14:24:10.408556 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:10.408547 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f859590f-c6cb-4b6c-bfd2-ebde80002e81-workload-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg\" (UID: \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" Nov 03 14:24:10.408636 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:10.408587 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f859590f-c6cb-4b6c-bfd2-ebde80002e81-istiod-ca-cert\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg\" (UID: \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" Nov 03 14:24:10.408669 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:10.408643 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f859590f-c6cb-4b6c-bfd2-ebde80002e81-credential-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg\" (UID: \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" Nov 03 14:24:10.408725 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:10.408711 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f859590f-c6cb-4b6c-bfd2-ebde80002e81-istio-data\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg\" (UID: \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" Nov 03 14:24:10.509531 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:10.509492 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f859590f-c6cb-4b6c-bfd2-ebde80002e81-credential-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg\" (UID: \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" Nov 03 14:24:10.509986 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:10.509553 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f859590f-c6cb-4b6c-bfd2-ebde80002e81-istio-data\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg\" (UID: \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" Nov 03 14:24:10.509986 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:10.509587 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f859590f-c6cb-4b6c-bfd2-ebde80002e81-istio-token\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg\" (UID: \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" Nov 03 14:24:10.509986 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:10.509639 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f859590f-c6cb-4b6c-bfd2-ebde80002e81-istio-envoy\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg\" (UID: \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" Nov 03 14:24:10.509986 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:10.509682 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f859590f-c6cb-4b6c-bfd2-ebde80002e81-workload-certs\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg\" (UID: \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" Nov 03 14:24:10.509986 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:10.509705 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f859590f-c6cb-4b6c-bfd2-ebde80002e81-istio-podinfo\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg\" (UID: \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" Nov 03 14:24:10.509986 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:10.509793 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w2v7\" (UniqueName: \"kubernetes.io/projected/f859590f-c6cb-4b6c-bfd2-ebde80002e81-kube-api-access-8w2v7\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg\" (UID: \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" Nov 03 14:24:10.509986 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:10.509825 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f859590f-c6cb-4b6c-bfd2-ebde80002e81-workload-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg\" (UID: \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" Nov 03 14:24:10.509986 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:10.509913 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f859590f-c6cb-4b6c-bfd2-ebde80002e81-istiod-ca-cert\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg\" (UID: \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" Nov 03 14:24:10.509986 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:10.509937 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f859590f-c6cb-4b6c-bfd2-ebde80002e81-credential-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg\" (UID: \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" Nov 03 14:24:10.510356 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:10.509994 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f859590f-c6cb-4b6c-bfd2-ebde80002e81-istio-data\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg\" (UID: \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" Nov 03 14:24:10.510480 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:10.510454 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f859590f-c6cb-4b6c-bfd2-ebde80002e81-workload-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg\" (UID: \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" Nov 03 14:24:10.510611 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:10.510496 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f859590f-c6cb-4b6c-bfd2-ebde80002e81-workload-certs\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg\" (UID: \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" Nov 03 14:24:10.510672 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:10.510616 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f859590f-c6cb-4b6c-bfd2-ebde80002e81-istiod-ca-cert\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg\" (UID: \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" Nov 03 14:24:10.512324 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:10.512307 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f859590f-c6cb-4b6c-bfd2-ebde80002e81-istio-envoy\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg\" (UID: \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" Nov 03 14:24:10.512493 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:10.512476 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f859590f-c6cb-4b6c-bfd2-ebde80002e81-istio-podinfo\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg\" (UID: \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" Nov 03 14:24:10.535283 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:10.535256 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f859590f-c6cb-4b6c-bfd2-ebde80002e81-istio-token\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg\" (UID: \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" Nov 03 14:24:10.535513 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:10.535495 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w2v7\" (UniqueName: \"kubernetes.io/projected/f859590f-c6cb-4b6c-bfd2-ebde80002e81-kube-api-access-8w2v7\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg\" (UID: \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" Nov 03 14:24:10.612687 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:10.612609 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" Nov 03 14:24:10.933220 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:10.933194 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg"] Nov 03 14:24:10.935516 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:24:10.935486 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf859590f_c6cb_4b6c_bfd2_ebde80002e81.slice/crio-c6b3ecc656f10b4a4187b29872d4a55aa200453ac873adf84a4e80295c503f6f WatchSource:0}: Error finding container c6b3ecc656f10b4a4187b29872d4a55aa200453ac873adf84a4e80295c503f6f: Status 404 returned error can't find the container with id c6b3ecc656f10b4a4187b29872d4a55aa200453ac873adf84a4e80295c503f6f Nov 03 14:24:10.938274 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:10.938240 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:24:10.938345 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:10.938304 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:24:10.938382 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:10.938349 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:24:10.964813 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:10.964784 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" event={"ID":"f859590f-c6cb-4b6c-bfd2-ebde80002e81","Type":"ContainerStarted","Data":"c6b3ecc656f10b4a4187b29872d4a55aa200453ac873adf84a4e80295c503f6f"} Nov 03 14:24:11.974864 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:11.974795 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" event={"ID":"f859590f-c6cb-4b6c-bfd2-ebde80002e81","Type":"ContainerStarted","Data":"34b172cbeb89aca4f1e6ca29cce8e5cfad22ead649e482baf802a82bb6b1e173"} Nov 03 14:24:12.051546 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:12.051479 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" podStartSLOduration=2.051459794 podStartE2EDuration="2.051459794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:24:12.037521552 +0000 UTC m=+785.926311134" watchObservedRunningTime="2026-11-03 14:24:12.051459794 +0000 UTC m=+785.940249377" Nov 03 14:24:12.617934 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:12.617896 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" Nov 03 14:24:12.621960 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:12.621930 5060 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 14:24:12.622176 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:12.622145 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" podUID="f859590f-c6cb-4b6c-bfd2-ebde80002e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 14:24:13.617531 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:13.617090 5060 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 14:24:13.617531 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:13.617151 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" podUID="f859590f-c6cb-4b6c-bfd2-ebde80002e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 14:24:14.613217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:14.613179 5060 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 14:24:14.613533 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:14.613500 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" podUID="f859590f-c6cb-4b6c-bfd2-ebde80002e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 14:24:15.613278 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:15.613235 5060 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 14:24:15.613930 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:15.613894 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" podUID="f859590f-c6cb-4b6c-bfd2-ebde80002e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 14:24:16.618231 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:16.618186 5060 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 14:24:16.618819 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:16.618788 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" podUID="f859590f-c6cb-4b6c-bfd2-ebde80002e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 14:24:17.947130 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:17.947095 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" Nov 03 14:24:17.949024 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:17.948998 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" Nov 03 14:24:17.950585 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:17.950561 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" Nov 03 14:24:21.714999 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:21.714966 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54c23ed0-5b8k2"] Nov 03 14:24:21.718806 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:21.718785 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c23ed0-5b8k2" Nov 03 14:24:21.734350 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:21.734328 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:24:21.744728 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:21.744696 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:24:21.744849 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:21.744739 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:24:21.744849 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:21.744814 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\"" Nov 03 14:24:21.748767 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:21.748742 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54c23ed0-5b8k2"] Nov 03 14:24:21.790573 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:21.790547 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2r6p\" (UniqueName: \"kubernetes.io/projected/de0fc154-3d30-47e8-b036-f65b4601ae9c-kube-api-access-p2r6p\") pod \"maas-api-revoke-sub-54c23ed0-5b8k2\" (UID: \"de0fc154-3d30-47e8-b036-f65b4601ae9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c23ed0-5b8k2" Nov 03 14:24:21.790721 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:21.790610 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de0fc154-3d30-47e8-b036-f65b4601ae9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54c23ed0-5b8k2\" (UID: \"de0fc154-3d30-47e8-b036-f65b4601ae9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c23ed0-5b8k2" Nov 03 14:24:21.790765 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:21.790724 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de0fc154-3d30-47e8-b036-f65b4601ae9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54c23ed0-5b8k2\" (UID: \"de0fc154-3d30-47e8-b036-f65b4601ae9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c23ed0-5b8k2" Nov 03 14:24:21.891941 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:21.891908 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2r6p\" (UniqueName: \"kubernetes.io/projected/de0fc154-3d30-47e8-b036-f65b4601ae9c-kube-api-access-p2r6p\") pod \"maas-api-revoke-sub-54c23ed0-5b8k2\" (UID: \"de0fc154-3d30-47e8-b036-f65b4601ae9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c23ed0-5b8k2" Nov 03 14:24:21.892115 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:21.892001 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de0fc154-3d30-47e8-b036-f65b4601ae9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54c23ed0-5b8k2\" (UID: \"de0fc154-3d30-47e8-b036-f65b4601ae9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c23ed0-5b8k2" Nov 03 14:24:21.892115 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:21.892060 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de0fc154-3d30-47e8-b036-f65b4601ae9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54c23ed0-5b8k2\" (UID: \"de0fc154-3d30-47e8-b036-f65b4601ae9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c23ed0-5b8k2" Nov 03 14:24:21.892882 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:21.892855 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de0fc154-3d30-47e8-b036-f65b4601ae9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54c23ed0-5b8k2\" (UID: \"de0fc154-3d30-47e8-b036-f65b4601ae9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c23ed0-5b8k2" Nov 03 14:24:21.894416 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:21.894389 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de0fc154-3d30-47e8-b036-f65b4601ae9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54c23ed0-5b8k2\" (UID: \"de0fc154-3d30-47e8-b036-f65b4601ae9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c23ed0-5b8k2" Nov 03 14:24:21.916038 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:21.916016 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2r6p\" (UniqueName: \"kubernetes.io/projected/de0fc154-3d30-47e8-b036-f65b4601ae9c-kube-api-access-p2r6p\") pod \"maas-api-revoke-sub-54c23ed0-5b8k2\" (UID: \"de0fc154-3d30-47e8-b036-f65b4601ae9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c23ed0-5b8k2" Nov 03 14:24:22.032925 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:22.032888 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c23ed0-5b8k2" Nov 03 14:24:22.343041 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:22.343014 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54c23ed0-5b8k2"] Nov 03 14:24:22.344899 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:24:22.344874 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde0fc154_3d30_47e8_b036_f65b4601ae9c.slice/crio-9279e46ee47915c6123339448ec08ede657ac7b0a5d06ca9feaca022a9149f33 WatchSource:0}: Error finding container 9279e46ee47915c6123339448ec08ede657ac7b0a5d06ca9feaca022a9149f33: Status 404 returned error can't find the container with id 9279e46ee47915c6123339448ec08ede657ac7b0a5d06ca9feaca022a9149f33 Nov 03 14:24:23.072695 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:23.072658 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c23ed0-5b8k2" event={"ID":"de0fc154-3d30-47e8-b036-f65b4601ae9c","Type":"ContainerStarted","Data":"a970c8bdc1982d4e9df5fce8a1a610dbc13a8da263f5d9412d01d1b24908e023"} Nov 03 14:24:23.073127 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:23.072721 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c23ed0-5b8k2" event={"ID":"de0fc154-3d30-47e8-b036-f65b4601ae9c","Type":"ContainerStarted","Data":"9279e46ee47915c6123339448ec08ede657ac7b0a5d06ca9feaca022a9149f33"} Nov 03 14:24:23.115741 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:23.115690 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c23ed0-5b8k2" podStartSLOduration=2.11567564 podStartE2EDuration="2.11567564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:24:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:24:23.112223895 +0000 UTC m=+797.001013477" watchObservedRunningTime="2026-11-03 14:24:23.11567564 +0000 UTC m=+797.004465215" Nov 03 14:24:24.080516 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:24.080484 5060 generic.go:356] "Generic (PLEG): container finished" podID="de0fc154-3d30-47e8-b036-f65b4601ae9c" containerID="a970c8bdc1982d4e9df5fce8a1a610dbc13a8da263f5d9412d01d1b24908e023" exitCode=0 Nov 03 14:24:24.080949 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:24.080552 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c23ed0-5b8k2" event={"ID":"de0fc154-3d30-47e8-b036-f65b4601ae9c","Type":"ContainerDied","Data":"a970c8bdc1982d4e9df5fce8a1a610dbc13a8da263f5d9412d01d1b24908e023"} Nov 03 14:24:25.243195 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:25.243168 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c23ed0-5b8k2" Nov 03 14:24:25.369388 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:25.333131 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de0fc154-3d30-47e8-b036-f65b4601ae9c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de0fc154-3d30-47e8-b036-f65b4601ae9c-maas-api-xks-ca\") pod \"de0fc154-3d30-47e8-b036-f65b4601ae9c\" (UID: \"de0fc154-3d30-47e8-b036-f65b4601ae9c\") " Nov 03 14:24:25.369388 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:25.333246 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de0fc154-3d30-47e8-b036-f65b4601ae9c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de0fc154-3d30-47e8-b036-f65b4601ae9c-maas-api-openshift-ca\") pod \"de0fc154-3d30-47e8-b036-f65b4601ae9c\" (UID: \"de0fc154-3d30-47e8-b036-f65b4601ae9c\") " Nov 03 14:24:25.369388 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:25.333411 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de0fc154-3d30-47e8-b036-f65b4601ae9c-kube-api-access-p2r6p\" (UniqueName: \"kubernetes.io/projected/de0fc154-3d30-47e8-b036-f65b4601ae9c-kube-api-access-p2r6p\") pod \"de0fc154-3d30-47e8-b036-f65b4601ae9c\" (UID: \"de0fc154-3d30-47e8-b036-f65b4601ae9c\") " Nov 03 14:24:25.374350 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:25.374295 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de0fc154-3d30-47e8-b036-f65b4601ae9c-maas-api-openshift-ca" pod "de0fc154-3d30-47e8-b036-f65b4601ae9c" (UID: "de0fc154-3d30-47e8-b036-f65b4601ae9c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:24:25.396336 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:25.382238 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de0fc154-3d30-47e8-b036-f65b4601ae9c-maas-api-openshift-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:24:25.401642 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:25.401602 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de0fc154-3d30-47e8-b036-f65b4601ae9c-kube-api-access-p2r6p" pod "de0fc154-3d30-47e8-b036-f65b4601ae9c" (UID: "de0fc154-3d30-47e8-b036-f65b4601ae9c"). InnerVolumeSpecName "kube-api-access-p2r6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:24:25.403131 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:25.403086 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de0fc154-3d30-47e8-b036-f65b4601ae9c-maas-api-xks-ca" pod "de0fc154-3d30-47e8-b036-f65b4601ae9c" (UID: "de0fc154-3d30-47e8-b036-f65b4601ae9c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:24:25.483702 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:25.483664 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de0fc154-3d30-47e8-b036-f65b4601ae9c-maas-api-xks-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:24:25.483702 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:25.483704 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2r6p\" (UniqueName: \"kubernetes.io/projected/de0fc154-3d30-47e8-b036-f65b4601ae9c-kube-api-access-p2r6p\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:24:26.093727 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:26.093693 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c23ed0-5b8k2" event={"ID":"de0fc154-3d30-47e8-b036-f65b4601ae9c","Type":"ContainerDied","Data":"9279e46ee47915c6123339448ec08ede657ac7b0a5d06ca9feaca022a9149f33"} Nov 03 14:24:26.093957 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:26.093754 5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9279e46ee47915c6123339448ec08ede657ac7b0a5d06ca9feaca022a9149f33" Nov 03 14:24:26.093957 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:26.093702 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c23ed0-5b8k2" Nov 03 14:24:38.952537 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:24:38.952432 5060 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc60d54c8_1f41_4f9c_9401_63a07e12f664.slice/crio-1bd3a1b6dec9eb61ece036757e428efd0cfc141931c5ae8d2595dc0166c1c964.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc60d54c8_1f41_4f9c_9401_63a07e12f664.slice/crio-conmon-1bd3a1b6dec9eb61ece036757e428efd0cfc141931c5ae8d2595dc0166c1c964.scope\": RecentStats: unable to find data in memory cache]" Nov 03 14:24:39.481616 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:39.481566 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" event={"ID":"c60d54c8-1f41-4f9c-9401-63a07e12f664","Type":"ContainerDied","Data":"1bd3a1b6dec9eb61ece036757e428efd0cfc141931c5ae8d2595dc0166c1c964"} Nov 03 14:24:39.482766 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:39.481524 5060 generic.go:356] "Generic (PLEG): container finished" podID="c60d54c8-1f41-4f9c-9401-63a07e12f664" containerID="1bd3a1b6dec9eb61ece036757e428efd0cfc141931c5ae8d2595dc0166c1c964" exitCode=0 Nov 03 14:24:39.483328 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:39.483307 5060 scope.go:122] "RemoveContainer" containerID="1bd3a1b6dec9eb61ece036757e428efd0cfc141931c5ae8d2595dc0166c1c964" Nov 03 14:24:39.486419 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:39.486383 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:24:39.486635 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:39.486591 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:24:39.486933 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:39.486909 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:24:39.487735 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:24:39.487718 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:24:40.490436 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:40.490395 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" event={"ID":"c60d54c8-1f41-4f9c-9401-63a07e12f664","Type":"ContainerStarted","Data":"fab7c2a973e48ae33680ceaa5e3a19186522999f512ae1903ced1b5d1cb65432"} Nov 03 14:24:40.827311 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:40.827226 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" Nov 03 14:24:40.828212 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:40.828186 5060 patch_prober.go:29] interesting pod/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 03 14:24:40.828334 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:40.828244 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" podUID="c60d54c8-1f41-4f9c-9401-63a07e12f664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 03 14:24:41.828869 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:41.828227 5060 patch_prober.go:29] interesting pod/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 03 14:24:41.828869 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:41.828309 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" podUID="c60d54c8-1f41-4f9c-9401-63a07e12f664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 03 14:24:42.828253 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:42.828209 5060 patch_prober.go:29] interesting pod/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 03 14:24:42.828481 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:42.828278 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" podUID="c60d54c8-1f41-4f9c-9401-63a07e12f664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 03 14:24:43.828252 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:43.828217 5060 patch_prober.go:29] interesting pod/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 03 14:24:43.828739 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:43.828277 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" podUID="c60d54c8-1f41-4f9c-9401-63a07e12f664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 03 14:24:43.878348 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:43.878317 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" Nov 03 14:24:44.828388 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:44.828347 5060 patch_prober.go:29] interesting pod/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 03 14:24:44.828927 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:44.828421 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" podUID="c60d54c8-1f41-4f9c-9401-63a07e12f664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 03 14:24:46.066044 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:46.066006 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" Nov 03 14:24:46.068913 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:46.068885 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" Nov 03 14:24:51.570860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:51.569915 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6e1e5a0-ng7sx"] Nov 03 14:24:51.570860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:51.570701 5060 cpu_manager.go:414] "containerMap: removing container" podUID="de0fc154-3d30-47e8-b036-f65b4601ae9c" containerName="revoke-keys" Nov 03 14:24:51.570860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:51.570724 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de0fc154-3d30-47e8-b036-f65b4601ae9c" containerName="revoke-keys" Nov 03 14:24:51.571615 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:51.570919 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="de0fc154-3d30-47e8-b036-f65b4601ae9c" containerName="revoke-keys" Nov 03 14:24:51.651888 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:51.636875 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e1e5a0-ng7sx" Nov 03 14:24:51.675857 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:51.669369 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:24:51.675857 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:51.669479 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:24:51.675857 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:51.669523 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:24:51.675857 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:51.669789 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\"" Nov 03 14:24:51.677023 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:51.676992 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6e1e5a0-ng7sx"] Nov 03 14:24:51.834560 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:51.834466 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeae7683-bed5-41bb-addf-27c59d5f2860-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6e1e5a0-ng7sx\" (UID: \"eeae7683-bed5-41bb-addf-27c59d5f2860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e1e5a0-ng7sx" Nov 03 14:24:51.857860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:51.834886 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeae7683-bed5-41bb-addf-27c59d5f2860-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6e1e5a0-ng7sx\" (UID: \"eeae7683-bed5-41bb-addf-27c59d5f2860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e1e5a0-ng7sx" Nov 03 14:24:51.894119 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:51.894077 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwpjn\" (UniqueName: \"kubernetes.io/projected/eeae7683-bed5-41bb-addf-27c59d5f2860-kube-api-access-qwpjn\") pod \"maas-api-revoke-sub-d6e1e5a0-ng7sx\" (UID: \"eeae7683-bed5-41bb-addf-27c59d5f2860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e1e5a0-ng7sx" Nov 03 14:24:52.048522 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:52.048474 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeae7683-bed5-41bb-addf-27c59d5f2860-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6e1e5a0-ng7sx\" (UID: \"eeae7683-bed5-41bb-addf-27c59d5f2860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e1e5a0-ng7sx" Nov 03 14:24:52.048934 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:52.048908 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwpjn\" (UniqueName: \"kubernetes.io/projected/eeae7683-bed5-41bb-addf-27c59d5f2860-kube-api-access-qwpjn\") pod \"maas-api-revoke-sub-d6e1e5a0-ng7sx\" (UID: \"eeae7683-bed5-41bb-addf-27c59d5f2860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e1e5a0-ng7sx" Nov 03 14:24:52.049163 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:52.049145 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeae7683-bed5-41bb-addf-27c59d5f2860-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6e1e5a0-ng7sx\" (UID: \"eeae7683-bed5-41bb-addf-27c59d5f2860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e1e5a0-ng7sx" Nov 03 14:24:52.050127 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:52.050105 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeae7683-bed5-41bb-addf-27c59d5f2860-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6e1e5a0-ng7sx\" (UID: \"eeae7683-bed5-41bb-addf-27c59d5f2860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e1e5a0-ng7sx" Nov 03 14:24:52.090962 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:52.085646 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwpjn\" (UniqueName: \"kubernetes.io/projected/eeae7683-bed5-41bb-addf-27c59d5f2860-kube-api-access-qwpjn\") pod \"maas-api-revoke-sub-d6e1e5a0-ng7sx\" (UID: \"eeae7683-bed5-41bb-addf-27c59d5f2860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e1e5a0-ng7sx" Nov 03 14:24:52.092315 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:52.092276 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeae7683-bed5-41bb-addf-27c59d5f2860-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6e1e5a0-ng7sx\" (UID: \"eeae7683-bed5-41bb-addf-27c59d5f2860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e1e5a0-ng7sx" Nov 03 14:24:52.111859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:52.106021 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e1e5a0-ng7sx" Nov 03 14:24:53.049093 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:53.047060 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6e1e5a0-ng7sx"] Nov 03 14:24:53.853809 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:53.853771 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e1e5a0-ng7sx" event={"ID":"eeae7683-bed5-41bb-addf-27c59d5f2860","Type":"ContainerStarted","Data":"e3987d59782d17e0e42aa24b981c6a532507b921df99ff07a73cb159792c907d"} Nov 03 14:24:53.854121 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:53.854096 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e1e5a0-ng7sx" event={"ID":"eeae7683-bed5-41bb-addf-27c59d5f2860","Type":"ContainerStarted","Data":"3e22e7113f12f38a7cd7122ed85c31d5011121e876fd530841151bb08729879c"} Nov 03 14:24:53.895355 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:53.895297 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e1e5a0-ng7sx" podStartSLOduration=2.895282502 podStartE2EDuration="2.895282502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:24:53.89352362 +0000 UTC m=+827.782313190" watchObservedRunningTime="2026-11-03 14:24:53.895282502 +0000 UTC m=+827.784072082" Nov 03 14:24:54.861708 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:54.861669 5060 generic.go:356] "Generic (PLEG): container finished" podID="eeae7683-bed5-41bb-addf-27c59d5f2860" containerID="e3987d59782d17e0e42aa24b981c6a532507b921df99ff07a73cb159792c907d" exitCode=0 Nov 03 14:24:54.862142 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:54.861751 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e1e5a0-ng7sx" event={"ID":"eeae7683-bed5-41bb-addf-27c59d5f2860","Type":"ContainerDied","Data":"e3987d59782d17e0e42aa24b981c6a532507b921df99ff07a73cb159792c907d"} Nov 03 14:24:56.013335 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:56.013310 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e1e5a0-ng7sx" Nov 03 14:24:56.100249 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:56.100220 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eeae7683-bed5-41bb-addf-27c59d5f2860-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeae7683-bed5-41bb-addf-27c59d5f2860-maas-api-openshift-ca\") pod \"eeae7683-bed5-41bb-addf-27c59d5f2860\" (UID: \"eeae7683-bed5-41bb-addf-27c59d5f2860\") " Nov 03 14:24:56.100457 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:56.100317 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eeae7683-bed5-41bb-addf-27c59d5f2860-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeae7683-bed5-41bb-addf-27c59d5f2860-maas-api-xks-ca\") pod \"eeae7683-bed5-41bb-addf-27c59d5f2860\" (UID: \"eeae7683-bed5-41bb-addf-27c59d5f2860\") " Nov 03 14:24:56.100660 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:56.100633 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeae7683-bed5-41bb-addf-27c59d5f2860-maas-api-openshift-ca" pod "eeae7683-bed5-41bb-addf-27c59d5f2860" (UID: "eeae7683-bed5-41bb-addf-27c59d5f2860"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:24:56.102452 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:56.102430 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeae7683-bed5-41bb-addf-27c59d5f2860-maas-api-xks-ca" pod "eeae7683-bed5-41bb-addf-27c59d5f2860" (UID: "eeae7683-bed5-41bb-addf-27c59d5f2860"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:24:56.201582 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:56.201490 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eeae7683-bed5-41bb-addf-27c59d5f2860-kube-api-access-qwpjn\" (UniqueName: \"kubernetes.io/projected/eeae7683-bed5-41bb-addf-27c59d5f2860-kube-api-access-qwpjn\") pod \"eeae7683-bed5-41bb-addf-27c59d5f2860\" (UID: \"eeae7683-bed5-41bb-addf-27c59d5f2860\") " Nov 03 14:24:56.201994 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:56.201974 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeae7683-bed5-41bb-addf-27c59d5f2860-maas-api-xks-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:24:56.202085 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:56.202013 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeae7683-bed5-41bb-addf-27c59d5f2860-maas-api-openshift-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:24:56.203386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:56.203356 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeae7683-bed5-41bb-addf-27c59d5f2860-kube-api-access-qwpjn" pod "eeae7683-bed5-41bb-addf-27c59d5f2860" (UID: "eeae7683-bed5-41bb-addf-27c59d5f2860"). InnerVolumeSpecName "kube-api-access-qwpjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:24:56.302950 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:56.302906 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwpjn\" (UniqueName: \"kubernetes.io/projected/eeae7683-bed5-41bb-addf-27c59d5f2860-kube-api-access-qwpjn\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:24:56.875910 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:56.875883 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e1e5a0-ng7sx" Nov 03 14:24:56.876102 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:56.875873 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e1e5a0-ng7sx" event={"ID":"eeae7683-bed5-41bb-addf-27c59d5f2860","Type":"ContainerDied","Data":"3e22e7113f12f38a7cd7122ed85c31d5011121e876fd530841151bb08729879c"} Nov 03 14:24:56.876102 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:56.876007 5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e22e7113f12f38a7cd7122ed85c31d5011121e876fd530841151bb08729879c" Nov 03 14:24:57.073500 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:57.073462 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7322-8e57e7"] Nov 03 14:24:57.074224 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:57.074191 5060 cpu_manager.go:414] "containerMap: removing container" podUID="eeae7683-bed5-41bb-addf-27c59d5f2860" containerName="revoke-keys" Nov 03 14:24:57.074224 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:57.074222 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eeae7683-bed5-41bb-addf-27c59d5f2860" containerName="revoke-keys" Nov 03 14:24:57.074425 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:57.074400 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="eeae7683-bed5-41bb-addf-27c59d5f2860" containerName="revoke-keys" Nov 03 14:24:57.078582 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:57.078563 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-8e57e7" Nov 03 14:24:57.095860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:57.095805 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-9sm86\"" Nov 03 14:24:57.098534 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:57.098511 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7322-8e57e7"] Nov 03 14:24:57.111585 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:57.111556 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b95dr\" (UniqueName: \"kubernetes.io/projected/3059872b-2687-4dab-9639-f4c8649d2102-kube-api-access-b95dr\") pod \"test-curl-7322-8e57e7\" (UID: \"3059872b-2687-4dab-9639-f4c8649d2102\") " pod="opendatahub/test-curl-7322-8e57e7" Nov 03 14:24:57.213099 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:57.213008 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b95dr\" (UniqueName: \"kubernetes.io/projected/3059872b-2687-4dab-9639-f4c8649d2102-kube-api-access-b95dr\") pod \"test-curl-7322-8e57e7\" (UID: \"3059872b-2687-4dab-9639-f4c8649d2102\") " pod="opendatahub/test-curl-7322-8e57e7" Nov 03 14:24:57.233916 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:57.233880 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b95dr\" (UniqueName: \"kubernetes.io/projected/3059872b-2687-4dab-9639-f4c8649d2102-kube-api-access-b95dr\") pod \"test-curl-7322-8e57e7\" (UID: \"3059872b-2687-4dab-9639-f4c8649d2102\") " pod="opendatahub/test-curl-7322-8e57e7" Nov 03 14:24:57.391041 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:57.391006 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-8e57e7" Nov 03 14:24:57.701408 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:57.701376 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7322-8e57e7"] Nov 03 14:24:57.702962 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:24:57.702931 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3059872b_2687_4dab_9639_f4c8649d2102.slice/crio-5d63ffbc9e432bfa4ccb38e0d71085853b3fba6eb06a119027af2aff0f31af59 WatchSource:0}: Error finding container 5d63ffbc9e432bfa4ccb38e0d71085853b3fba6eb06a119027af2aff0f31af59: Status 404 returned error can't find the container with id 5d63ffbc9e432bfa4ccb38e0d71085853b3fba6eb06a119027af2aff0f31af59 Nov 03 14:24:57.884012 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:57.883973 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7322-8e57e7" event={"ID":"3059872b-2687-4dab-9639-f4c8649d2102","Type":"ContainerStarted","Data":"e2e4458ad7a4c62b063e458d85b7fd465c811d3d1f9f4fd1c277990c256f44ad"} Nov 03 14:24:57.884012 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:57.884015 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7322-8e57e7" event={"ID":"3059872b-2687-4dab-9639-f4c8649d2102","Type":"ContainerStarted","Data":"5d63ffbc9e432bfa4ccb38e0d71085853b3fba6eb06a119027af2aff0f31af59"} Nov 03 14:24:57.924635 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:57.924577 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7322-8e57e7" podStartSLOduration=0.924561456 podStartE2EDuration="924.561456ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:24:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:24:57.922151722 +0000 UTC m=+831.810941304" watchObservedRunningTime="2026-11-03 14:24:57.924561456 +0000 UTC m=+831.813351038" Nov 03 14:24:58.776510 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:58.776474 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7322-8e57e7"] Nov 03 14:24:58.781465 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:58.781434 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7322-8e57e7"] Nov 03 14:24:59.742481 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:59.742441 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8116d673-lnczp"] Nov 03 14:24:59.743190 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:59.743170 5060 cpu_manager.go:414] "containerMap: removing container" podUID="3059872b-2687-4dab-9639-f4c8649d2102" containerName="test-curl-7322-8e57e7" Nov 03 14:24:59.743190 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:59.743190 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3059872b-2687-4dab-9639-f4c8649d2102" containerName="test-curl-7322-8e57e7" Nov 03 14:24:59.743367 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:59.743311 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="3059872b-2687-4dab-9639-f4c8649d2102" containerName="test-curl-7322-8e57e7" Nov 03 14:24:59.747203 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:59.747182 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8116d673-lnczp" Nov 03 14:24:59.768514 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:59.768486 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8116d673-lnczp"] Nov 03 14:24:59.772974 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:59.772954 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:24:59.773104 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:59.772961 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:24:59.773104 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:59.772991 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:24:59.773104 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:59.773015 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\"" Nov 03 14:24:59.852174 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:59.852139 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06a09272-2e46-4ee6-b035-74cea0c54feb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8116d673-lnczp\" (UID: \"06a09272-2e46-4ee6-b035-74cea0c54feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8116d673-lnczp" Nov 03 14:24:59.852551 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:59.852313 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55vzp\" (UniqueName: \"kubernetes.io/projected/06a09272-2e46-4ee6-b035-74cea0c54feb-kube-api-access-55vzp\") pod \"maas-api-revoke-sub-8116d673-lnczp\" (UID: \"06a09272-2e46-4ee6-b035-74cea0c54feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8116d673-lnczp" Nov 03 14:24:59.852551 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:59.852349 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06a09272-2e46-4ee6-b035-74cea0c54feb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8116d673-lnczp\" (UID: \"06a09272-2e46-4ee6-b035-74cea0c54feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8116d673-lnczp" Nov 03 14:24:59.897235 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:59.897182 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7322-8e57e7" podUID="3059872b-2687-4dab-9639-f4c8649d2102" containerName="test-curl-7322-8e57e7" containerID="cri-o://e2e4458ad7a4c62b063e458d85b7fd465c811d3d1f9f4fd1c277990c256f44ad" gracePeriod=2 Nov 03 14:24:59.909696 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:24:59.909646 5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7322-8e57e7\" is forbidden: User \"system:node:ip-10-0-143-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-171.ec2.internal' and this object" podUID="3059872b-2687-4dab-9639-f4c8649d2102" pod="opendatahub/test-curl-7322-8e57e7" Nov 03 14:24:59.953973 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:59.953937 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06a09272-2e46-4ee6-b035-74cea0c54feb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8116d673-lnczp\" (UID: \"06a09272-2e46-4ee6-b035-74cea0c54feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8116d673-lnczp" Nov 03 14:24:59.954147 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:59.954055 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55vzp\" (UniqueName: \"kubernetes.io/projected/06a09272-2e46-4ee6-b035-74cea0c54feb-kube-api-access-55vzp\") pod \"maas-api-revoke-sub-8116d673-lnczp\" (UID: \"06a09272-2e46-4ee6-b035-74cea0c54feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8116d673-lnczp" Nov 03 14:24:59.954147 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:59.954082 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06a09272-2e46-4ee6-b035-74cea0c54feb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8116d673-lnczp\" (UID: \"06a09272-2e46-4ee6-b035-74cea0c54feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8116d673-lnczp" Nov 03 14:24:59.954747 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:59.954723 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06a09272-2e46-4ee6-b035-74cea0c54feb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8116d673-lnczp\" (UID: \"06a09272-2e46-4ee6-b035-74cea0c54feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8116d673-lnczp" Nov 03 14:24:59.956366 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:59.956348 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06a09272-2e46-4ee6-b035-74cea0c54feb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8116d673-lnczp\" (UID: \"06a09272-2e46-4ee6-b035-74cea0c54feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8116d673-lnczp" Nov 03 14:24:59.975704 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:24:59.975677 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55vzp\" (UniqueName: \"kubernetes.io/projected/06a09272-2e46-4ee6-b035-74cea0c54feb-kube-api-access-55vzp\") pod \"maas-api-revoke-sub-8116d673-lnczp\" (UID: \"06a09272-2e46-4ee6-b035-74cea0c54feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8116d673-lnczp" Nov 03 14:25:00.060145 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:00.060067 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8116d673-lnczp" Nov 03 14:25:00.375126 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:00.375096 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8116d673-lnczp"] Nov 03 14:25:00.375976 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:25:00.375948 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06a09272_2e46_4ee6_b035_74cea0c54feb.slice/crio-bc69433bf75930dca7cc9581333874e7001a309bfe9baec63c5f125c01190d14 WatchSource:0}: Error finding container bc69433bf75930dca7cc9581333874e7001a309bfe9baec63c5f125c01190d14: Status 404 returned error can't find the container with id bc69433bf75930dca7cc9581333874e7001a309bfe9baec63c5f125c01190d14 Nov 03 14:25:00.921858 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:00.921114 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8116d673-lnczp" event={"ID":"06a09272-2e46-4ee6-b035-74cea0c54feb","Type":"ContainerStarted","Data":"0d62004ea2b310d6b52c8361502c989dab4034e3a0aa26f347dc4ca8bf46238f"} Nov 03 14:25:00.921858 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:00.921182 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8116d673-lnczp" event={"ID":"06a09272-2e46-4ee6-b035-74cea0c54feb","Type":"ContainerStarted","Data":"bc69433bf75930dca7cc9581333874e7001a309bfe9baec63c5f125c01190d14"} Nov 03 14:25:01.990863 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:01.987539 5060 generic.go:356] "Generic (PLEG): container finished" podID="06a09272-2e46-4ee6-b035-74cea0c54feb" containerID="0d62004ea2b310d6b52c8361502c989dab4034e3a0aa26f347dc4ca8bf46238f" exitCode=0 Nov 03 14:25:01.990863 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:01.987800 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8116d673-lnczp" event={"ID":"06a09272-2e46-4ee6-b035-74cea0c54feb","Type":"ContainerDied","Data":"0d62004ea2b310d6b52c8361502c989dab4034e3a0aa26f347dc4ca8bf46238f"} Nov 03 14:25:02.685218 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:02.685054 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-8e57e7" Nov 03 14:25:02.789626 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:02.789592 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3059872b-2687-4dab-9639-f4c8649d2102-kube-api-access-b95dr\" (UniqueName: \"kubernetes.io/projected/3059872b-2687-4dab-9639-f4c8649d2102-kube-api-access-b95dr\") pod \"3059872b-2687-4dab-9639-f4c8649d2102\" (UID: \"3059872b-2687-4dab-9639-f4c8649d2102\") " Nov 03 14:25:02.792015 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:02.791976 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3059872b-2687-4dab-9639-f4c8649d2102-kube-api-access-b95dr" pod "3059872b-2687-4dab-9639-f4c8649d2102" (UID: "3059872b-2687-4dab-9639-f4c8649d2102"). InnerVolumeSpecName "kube-api-access-b95dr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:25:02.891428 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:02.891333 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b95dr\" (UniqueName: \"kubernetes.io/projected/3059872b-2687-4dab-9639-f4c8649d2102-kube-api-access-b95dr\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:25:02.997671 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:02.996908 5060 generic.go:356] "Generic (PLEG): container finished" podID="3059872b-2687-4dab-9639-f4c8649d2102" containerID="e2e4458ad7a4c62b063e458d85b7fd465c811d3d1f9f4fd1c277990c256f44ad" exitCode=137 Nov 03 14:25:02.997671 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:02.997365 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-8e57e7" Nov 03 14:25:02.998490 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:02.998288 5060 scope.go:122] "RemoveContainer" containerID="e2e4458ad7a4c62b063e458d85b7fd465c811d3d1f9f4fd1c277990c256f44ad" Nov 03 14:25:03.034160 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:25:03.034097 5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7322-8e57e7\" is forbidden: User \"system:node:ip-10-0-143-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-171.ec2.internal' and this object" podUID="3059872b-2687-4dab-9639-f4c8649d2102" pod="opendatahub/test-curl-7322-8e57e7" Nov 03 14:25:03.057190 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:03.049442 5060 scope.go:122] "RemoveContainer" containerID="e2e4458ad7a4c62b063e458d85b7fd465c811d3d1f9f4fd1c277990c256f44ad" Nov 03 14:25:03.057190 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:25:03.050003 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2e4458ad7a4c62b063e458d85b7fd465c811d3d1f9f4fd1c277990c256f44ad\": container with ID starting with e2e4458ad7a4c62b063e458d85b7fd465c811d3d1f9f4fd1c277990c256f44ad not found: ID does not exist" containerID="e2e4458ad7a4c62b063e458d85b7fd465c811d3d1f9f4fd1c277990c256f44ad" Nov 03 14:25:03.057190 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:03.050040 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2e4458ad7a4c62b063e458d85b7fd465c811d3d1f9f4fd1c277990c256f44ad"} err="failed to get container status \"e2e4458ad7a4c62b063e458d85b7fd465c811d3d1f9f4fd1c277990c256f44ad\": rpc error: code = NotFound desc = could not find container \"e2e4458ad7a4c62b063e458d85b7fd465c811d3d1f9f4fd1c277990c256f44ad\": container with ID starting with e2e4458ad7a4c62b063e458d85b7fd465c811d3d1f9f4fd1c277990c256f44ad not found: ID does not exist" Nov 03 14:25:03.390332 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:03.390100 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8116d673-lnczp" Nov 03 14:25:03.408032 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:03.407974 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06a09272-2e46-4ee6-b035-74cea0c54feb-kube-api-access-55vzp\" (UniqueName: \"kubernetes.io/projected/06a09272-2e46-4ee6-b035-74cea0c54feb-kube-api-access-55vzp\") pod \"06a09272-2e46-4ee6-b035-74cea0c54feb\" (UID: \"06a09272-2e46-4ee6-b035-74cea0c54feb\") " Nov 03 14:25:03.408225 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:03.408068 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06a09272-2e46-4ee6-b035-74cea0c54feb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06a09272-2e46-4ee6-b035-74cea0c54feb-maas-api-openshift-ca\") pod \"06a09272-2e46-4ee6-b035-74cea0c54feb\" (UID: \"06a09272-2e46-4ee6-b035-74cea0c54feb\") " Nov 03 14:25:03.408225 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:03.408118 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06a09272-2e46-4ee6-b035-74cea0c54feb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06a09272-2e46-4ee6-b035-74cea0c54feb-maas-api-xks-ca\") pod \"06a09272-2e46-4ee6-b035-74cea0c54feb\" (UID: \"06a09272-2e46-4ee6-b035-74cea0c54feb\") " Nov 03 14:25:03.408885 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:03.408856 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06a09272-2e46-4ee6-b035-74cea0c54feb-maas-api-openshift-ca" pod "06a09272-2e46-4ee6-b035-74cea0c54feb" (UID: "06a09272-2e46-4ee6-b035-74cea0c54feb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:25:03.409234 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:03.409212 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06a09272-2e46-4ee6-b035-74cea0c54feb-maas-api-openshift-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:25:03.411120 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:03.411091 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06a09272-2e46-4ee6-b035-74cea0c54feb-maas-api-xks-ca" pod "06a09272-2e46-4ee6-b035-74cea0c54feb" (UID: "06a09272-2e46-4ee6-b035-74cea0c54feb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:25:03.411238 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:03.411092 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06a09272-2e46-4ee6-b035-74cea0c54feb-kube-api-access-55vzp" pod "06a09272-2e46-4ee6-b035-74cea0c54feb" (UID: "06a09272-2e46-4ee6-b035-74cea0c54feb"). InnerVolumeSpecName "kube-api-access-55vzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:25:03.443765 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:25:03.443720 5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7322-8e57e7\" is forbidden: User \"system:node:ip-10-0-143-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-171.ec2.internal' and this object" podUID="3059872b-2687-4dab-9639-f4c8649d2102" pod="opendatahub/test-curl-7322-8e57e7" Nov 03 14:25:03.513865 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:03.512748 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06a09272-2e46-4ee6-b035-74cea0c54feb-maas-api-xks-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:25:03.513865 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:03.512795 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55vzp\" (UniqueName: \"kubernetes.io/projected/06a09272-2e46-4ee6-b035-74cea0c54feb-kube-api-access-55vzp\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:25:04.004392 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:04.004354 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8116d673-lnczp" Nov 03 14:25:04.004862 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:04.004425 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8116d673-lnczp" event={"ID":"06a09272-2e46-4ee6-b035-74cea0c54feb","Type":"ContainerDied","Data":"bc69433bf75930dca7cc9581333874e7001a309bfe9baec63c5f125c01190d14"} Nov 03 14:25:04.004862 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:04.004474 5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc69433bf75930dca7cc9581333874e7001a309bfe9baec63c5f125c01190d14" Nov 03 14:25:04.040269 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:25:04.040225 5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7322-8e57e7\" is forbidden: User \"system:node:ip-10-0-143-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-171.ec2.internal' and this object" podUID="3059872b-2687-4dab-9639-f4c8649d2102" pod="opendatahub/test-curl-7322-8e57e7" Nov 03 14:25:04.250113 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:04.250067 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3059872b-2687-4dab-9639-f4c8649d2102" path="/var/lib/kubelet/pods/3059872b-2687-4dab-9639-f4c8649d2102/volumes" Nov 03 14:25:09.490230 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:25:09.490190 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:25:14.608243 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:14.608207 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4"] Nov 03 14:25:14.608932 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:14.608911 5060 cpu_manager.go:414] "containerMap: removing container" podUID="06a09272-2e46-4ee6-b035-74cea0c54feb" containerName="revoke-keys" Nov 03 14:25:14.609035 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:14.608933 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06a09272-2e46-4ee6-b035-74cea0c54feb" containerName="revoke-keys" Nov 03 14:25:14.609098 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:14.609086 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="06a09272-2e46-4ee6-b035-74cea0c54feb" containerName="revoke-keys" Nov 03 14:25:14.613882 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:14.613859 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" Nov 03 14:25:14.629323 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:14.629298 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-2d0da2-openshift-default-dockercfg-288bc\"" Nov 03 14:25:14.644158 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:14.644129 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4"] Nov 03 14:25:14.786419 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:14.786390 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-istio-envoy\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4\" (UID: \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" Nov 03 14:25:14.786419 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:14.786428 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-istio-token\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4\" (UID: \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" Nov 03 14:25:14.786653 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:14.786624 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-workload-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4\" (UID: \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" Nov 03 14:25:14.786712 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:14.786666 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-istio-data\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4\" (UID: \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" Nov 03 14:25:14.786712 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:14.786688 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-credential-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4\" (UID: \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" Nov 03 14:25:14.786786 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:14.786765 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-istio-podinfo\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4\" (UID: \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" Nov 03 14:25:14.786786 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:14.786782 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-istiod-ca-cert\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4\" (UID: \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" Nov 03 14:25:14.786899 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:14.786875 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwk5n\" (UniqueName: \"kubernetes.io/projected/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-kube-api-access-cwk5n\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4\" (UID: \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" Nov 03 14:25:14.786955 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:14.786910 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-workload-certs\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4\" (UID: \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" Nov 03 14:25:14.888147 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:14.888058 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-workload-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4\" (UID: \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" Nov 03 14:25:14.888147 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:14.888101 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-istio-data\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4\" (UID: \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" Nov 03 14:25:14.888147 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:14.888120 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-credential-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4\" (UID: \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" Nov 03 14:25:14.888436 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:14.888157 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-istio-podinfo\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4\" (UID: \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" Nov 03 14:25:14.888436 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:14.888173 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-istiod-ca-cert\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4\" (UID: \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" Nov 03 14:25:14.888436 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:14.888230 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwk5n\" (UniqueName: \"kubernetes.io/projected/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-kube-api-access-cwk5n\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4\" (UID: \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" Nov 03 14:25:14.888436 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:14.888266 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-workload-certs\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4\" (UID: \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" Nov 03 14:25:14.888436 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:14.888334 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-istio-envoy\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4\" (UID: \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" Nov 03 14:25:14.888436 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:14.888362 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-istio-token\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4\" (UID: \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" Nov 03 14:25:14.888744 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:14.888687 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-workload-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4\" (UID: \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" Nov 03 14:25:14.888786 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:14.888742 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-istio-data\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4\" (UID: \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" Nov 03 14:25:14.889077 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:14.889040 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-credential-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4\" (UID: \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" Nov 03 14:25:14.889211 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:14.889077 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-workload-certs\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4\" (UID: \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" Nov 03 14:25:14.889354 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:14.889333 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-istiod-ca-cert\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4\" (UID: \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" Nov 03 14:25:14.890751 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:14.890725 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-istio-envoy\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4\" (UID: \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" Nov 03 14:25:14.890827 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:14.890806 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-istio-podinfo\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4\" (UID: \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" Nov 03 14:25:14.909528 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:14.909507 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-istio-token\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4\" (UID: \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" Nov 03 14:25:14.909639 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:14.909622 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwk5n\" (UniqueName: \"kubernetes.io/projected/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-kube-api-access-cwk5n\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4\" (UID: \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" Nov 03 14:25:14.961642 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:14.961617 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" Nov 03 14:25:15.282899 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:15.282866 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4"] Nov 03 14:25:15.284386 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:25:15.284362 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab1a1c4a_c6c5_45bb_b3ee_529a355c3691.slice/crio-11f34292c1b8574acba8f59c4492146bcc9245b7efbed2f67e9c6412bcfcb409 WatchSource:0}: Error finding container 11f34292c1b8574acba8f59c4492146bcc9245b7efbed2f67e9c6412bcfcb409: Status 404 returned error can't find the container with id 11f34292c1b8574acba8f59c4492146bcc9245b7efbed2f67e9c6412bcfcb409 Nov 03 14:25:15.287023 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:15.286992 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:25:15.287107 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:15.287054 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:25:15.287107 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:15.287080 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:25:16.131778 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:16.131745 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" event={"ID":"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691","Type":"ContainerStarted","Data":"e8942975458ba43bb3ffab3ea5cde96288b5049eaa6cac0cb4797606df0c144c"} Nov 03 14:25:16.132309 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:16.131803 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" event={"ID":"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691","Type":"ContainerStarted","Data":"11f34292c1b8574acba8f59c4492146bcc9245b7efbed2f67e9c6412bcfcb409"} Nov 03 14:25:16.196470 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:16.196420 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" podStartSLOduration=2.196405432 podStartE2EDuration="2.196405432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:25:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:25:16.184720238 +0000 UTC m=+850.073509821" watchObservedRunningTime="2026-11-03 14:25:16.196405432 +0000 UTC m=+850.085195012" Nov 03 14:25:16.962660 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:16.962620 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" Nov 03 14:25:16.964253 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:16.964223 5060 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 14:25:16.964376 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:16.964275 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" podUID="ab1a1c4a-c6c5-45bb-b3ee-529a355c3691" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 14:25:16.978368 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:16.978340 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-866507d3-kkqds"] Nov 03 14:25:16.982517 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:16.982496 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866507d3-kkqds" Nov 03 14:25:17.017859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:17.014939 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\"" Nov 03 14:25:17.017859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:17.015054 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:25:17.017859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:17.015116 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:25:17.030057 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:17.030015 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:25:17.034070 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:17.034027 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-866507d3-kkqds"] Nov 03 14:25:17.037382 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:17.037353 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3be4cf1-f542-46ac-92e4-61140d21c04c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-866507d3-kkqds\" (UID: \"a3be4cf1-f542-46ac-92e4-61140d21c04c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866507d3-kkqds" Nov 03 14:25:17.037528 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:17.037401 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3be4cf1-f542-46ac-92e4-61140d21c04c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-866507d3-kkqds\" (UID: \"a3be4cf1-f542-46ac-92e4-61140d21c04c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866507d3-kkqds" Nov 03 14:25:17.037528 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:17.037491 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvzdq\" (UniqueName: \"kubernetes.io/projected/a3be4cf1-f542-46ac-92e4-61140d21c04c-kube-api-access-zvzdq\") pod \"maas-api-revoke-sub-866507d3-kkqds\" (UID: \"a3be4cf1-f542-46ac-92e4-61140d21c04c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866507d3-kkqds" Nov 03 14:25:17.138194 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:17.138136 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvzdq\" (UniqueName: \"kubernetes.io/projected/a3be4cf1-f542-46ac-92e4-61140d21c04c-kube-api-access-zvzdq\") pod \"maas-api-revoke-sub-866507d3-kkqds\" (UID: \"a3be4cf1-f542-46ac-92e4-61140d21c04c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866507d3-kkqds" Nov 03 14:25:17.138910 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:17.138885 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3be4cf1-f542-46ac-92e4-61140d21c04c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-866507d3-kkqds\" (UID: \"a3be4cf1-f542-46ac-92e4-61140d21c04c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866507d3-kkqds" Nov 03 14:25:17.139027 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:17.138925 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3be4cf1-f542-46ac-92e4-61140d21c04c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-866507d3-kkqds\" (UID: \"a3be4cf1-f542-46ac-92e4-61140d21c04c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866507d3-kkqds" Nov 03 14:25:17.139563 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:17.139545 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3be4cf1-f542-46ac-92e4-61140d21c04c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-866507d3-kkqds\" (UID: \"a3be4cf1-f542-46ac-92e4-61140d21c04c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866507d3-kkqds" Nov 03 14:25:17.141318 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:17.141297 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3be4cf1-f542-46ac-92e4-61140d21c04c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-866507d3-kkqds\" (UID: \"a3be4cf1-f542-46ac-92e4-61140d21c04c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866507d3-kkqds" Nov 03 14:25:17.159284 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:17.159258 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvzdq\" (UniqueName: \"kubernetes.io/projected/a3be4cf1-f542-46ac-92e4-61140d21c04c-kube-api-access-zvzdq\") pod \"maas-api-revoke-sub-866507d3-kkqds\" (UID: \"a3be4cf1-f542-46ac-92e4-61140d21c04c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866507d3-kkqds" Nov 03 14:25:17.296882 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:17.296813 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866507d3-kkqds" Nov 03 14:25:17.629980 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:17.629948 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-866507d3-kkqds"] Nov 03 14:25:17.631810 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:25:17.631783 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3be4cf1_f542_46ac_92e4_61140d21c04c.slice/crio-e7fc6f18b6158ebd8f109b7e070388139690fda89473683a85d1a3c5f1c5b990 WatchSource:0}: Error finding container e7fc6f18b6158ebd8f109b7e070388139690fda89473683a85d1a3c5f1c5b990: Status 404 returned error can't find the container with id e7fc6f18b6158ebd8f109b7e070388139690fda89473683a85d1a3c5f1c5b990 Nov 03 14:25:17.962828 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:17.962738 5060 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 14:25:17.962828 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:17.962803 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" podUID="ab1a1c4a-c6c5-45bb-b3ee-529a355c3691" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 14:25:18.145516 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:18.145479 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866507d3-kkqds" event={"ID":"a3be4cf1-f542-46ac-92e4-61140d21c04c","Type":"ContainerStarted","Data":"e3e8ed9dc643fac1c288c6330e254fb3152cda3b4dcb8df68efa6cb5ab0c4aca"} Nov 03 14:25:18.145958 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:18.145545 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866507d3-kkqds" event={"ID":"a3be4cf1-f542-46ac-92e4-61140d21c04c","Type":"ContainerStarted","Data":"e7fc6f18b6158ebd8f109b7e070388139690fda89473683a85d1a3c5f1c5b990"} Nov 03 14:25:18.187468 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:18.187416 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866507d3-kkqds" podStartSLOduration=2.187400893 podStartE2EDuration="2.187400893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:25:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:25:18.184720232 +0000 UTC m=+852.073509817" watchObservedRunningTime="2026-11-03 14:25:18.187400893 +0000 UTC m=+852.076190473" Nov 03 14:25:18.963111 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:18.963018 5060 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 14:25:18.963111 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:18.963072 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" podUID="ab1a1c4a-c6c5-45bb-b3ee-529a355c3691" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 14:25:19.154041 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:19.154004 5060 generic.go:356] "Generic (PLEG): container finished" podID="a3be4cf1-f542-46ac-92e4-61140d21c04c" containerID="e3e8ed9dc643fac1c288c6330e254fb3152cda3b4dcb8df68efa6cb5ab0c4aca" exitCode=0 Nov 03 14:25:19.154403 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:19.154082 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866507d3-kkqds" event={"ID":"a3be4cf1-f542-46ac-92e4-61140d21c04c","Type":"ContainerDied","Data":"e3e8ed9dc643fac1c288c6330e254fb3152cda3b4dcb8df68efa6cb5ab0c4aca"} Nov 03 14:25:19.962601 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:19.962554 5060 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 14:25:19.962788 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:19.962624 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" podUID="ab1a1c4a-c6c5-45bb-b3ee-529a355c3691" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 14:25:20.365275 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:20.365252 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866507d3-kkqds" Nov 03 14:25:20.473098 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:20.473048 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3be4cf1-f542-46ac-92e4-61140d21c04c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3be4cf1-f542-46ac-92e4-61140d21c04c-maas-api-openshift-ca\") pod \"a3be4cf1-f542-46ac-92e4-61140d21c04c\" (UID: \"a3be4cf1-f542-46ac-92e4-61140d21c04c\") " Nov 03 14:25:20.473260 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:20.473209 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3be4cf1-f542-46ac-92e4-61140d21c04c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3be4cf1-f542-46ac-92e4-61140d21c04c-maas-api-xks-ca\") pod \"a3be4cf1-f542-46ac-92e4-61140d21c04c\" (UID: \"a3be4cf1-f542-46ac-92e4-61140d21c04c\") " Nov 03 14:25:20.473260 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:20.473247 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3be4cf1-f542-46ac-92e4-61140d21c04c-kube-api-access-zvzdq\" (UniqueName: \"kubernetes.io/projected/a3be4cf1-f542-46ac-92e4-61140d21c04c-kube-api-access-zvzdq\") pod \"a3be4cf1-f542-46ac-92e4-61140d21c04c\" (UID: \"a3be4cf1-f542-46ac-92e4-61140d21c04c\") " Nov 03 14:25:20.473475 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:20.473444 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3be4cf1-f542-46ac-92e4-61140d21c04c-maas-api-openshift-ca" pod "a3be4cf1-f542-46ac-92e4-61140d21c04c" (UID: "a3be4cf1-f542-46ac-92e4-61140d21c04c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:25:20.473704 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:20.473678 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3be4cf1-f542-46ac-92e4-61140d21c04c-maas-api-openshift-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:25:20.476036 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:20.475997 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3be4cf1-f542-46ac-92e4-61140d21c04c-maas-api-xks-ca" pod "a3be4cf1-f542-46ac-92e4-61140d21c04c" (UID: "a3be4cf1-f542-46ac-92e4-61140d21c04c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:25:20.476137 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:20.476076 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3be4cf1-f542-46ac-92e4-61140d21c04c-kube-api-access-zvzdq" pod "a3be4cf1-f542-46ac-92e4-61140d21c04c" (UID: "a3be4cf1-f542-46ac-92e4-61140d21c04c"). InnerVolumeSpecName "kube-api-access-zvzdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:25:20.574820 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:20.574736 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3be4cf1-f542-46ac-92e4-61140d21c04c-maas-api-xks-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:25:20.574820 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:20.574768 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvzdq\" (UniqueName: \"kubernetes.io/projected/a3be4cf1-f542-46ac-92e4-61140d21c04c-kube-api-access-zvzdq\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:25:20.962784 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:20.962697 5060 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 14:25:20.962784 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:20.962753 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" podUID="ab1a1c4a-c6c5-45bb-b3ee-529a355c3691" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 14:25:21.170527 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:21.170497 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866507d3-kkqds" Nov 03 14:25:21.170690 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:21.170496 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866507d3-kkqds" event={"ID":"a3be4cf1-f542-46ac-92e4-61140d21c04c","Type":"ContainerDied","Data":"e7fc6f18b6158ebd8f109b7e070388139690fda89473683a85d1a3c5f1c5b990"} Nov 03 14:25:21.170690 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:21.170653 5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7fc6f18b6158ebd8f109b7e070388139690fda89473683a85d1a3c5f1c5b990" Nov 03 14:25:22.097048 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:22.097013 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" Nov 03 14:25:22.097801 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:22.097782 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" Nov 03 14:25:22.098632 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:22.098612 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" Nov 03 14:25:27.260592 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:27.260523 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7322-a13e07"] Nov 03 14:25:27.298858 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:27.291431 5060 cpu_manager.go:414] "containerMap: removing container" podUID="a3be4cf1-f542-46ac-92e4-61140d21c04c" containerName="revoke-keys" Nov 03 14:25:27.298858 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:27.291461 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3be4cf1-f542-46ac-92e4-61140d21c04c" containerName="revoke-keys" Nov 03 14:25:27.298858 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:27.291541 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3be4cf1-f542-46ac-92e4-61140d21c04c" containerName="revoke-keys" Nov 03 14:25:27.302875 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:27.302356 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7322-a13e07"] Nov 03 14:25:27.302875 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:27.302471 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-a13e07" Nov 03 14:25:27.324212 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:27.323464 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-9sm86\"" Nov 03 14:25:27.431451 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:27.426842 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm74x\" (UniqueName: \"kubernetes.io/projected/10f26a66-e9a1-4412-9362-b3e893b8c11b-kube-api-access-xm74x\") pod \"test-curl-7322-a13e07\" (UID: \"10f26a66-e9a1-4412-9362-b3e893b8c11b\") " pod="opendatahub/test-curl-7322-a13e07" Nov 03 14:25:27.528541 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:27.528451 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm74x\" (UniqueName: \"kubernetes.io/projected/10f26a66-e9a1-4412-9362-b3e893b8c11b-kube-api-access-xm74x\") pod \"test-curl-7322-a13e07\" (UID: \"10f26a66-e9a1-4412-9362-b3e893b8c11b\") " pod="opendatahub/test-curl-7322-a13e07" Nov 03 14:25:27.553898 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:27.553867 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm74x\" (UniqueName: \"kubernetes.io/projected/10f26a66-e9a1-4412-9362-b3e893b8c11b-kube-api-access-xm74x\") pod \"test-curl-7322-a13e07\" (UID: \"10f26a66-e9a1-4412-9362-b3e893b8c11b\") " pod="opendatahub/test-curl-7322-a13e07" Nov 03 14:25:27.631993 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:27.631960 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-a13e07" Nov 03 14:25:27.950858 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:27.950813 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7322-a13e07"] Nov 03 14:25:27.953268 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:25:27.953230 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10f26a66_e9a1_4412_9362_b3e893b8c11b.slice/crio-9716457158a0a000b1c37aec9b3647e17e8807fbb38ad65986f51539d2292f8d WatchSource:0}: Error finding container 9716457158a0a000b1c37aec9b3647e17e8807fbb38ad65986f51539d2292f8d: Status 404 returned error can't find the container with id 9716457158a0a000b1c37aec9b3647e17e8807fbb38ad65986f51539d2292f8d Nov 03 14:25:28.377690 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:28.377656 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7322-a13e07" event={"ID":"10f26a66-e9a1-4412-9362-b3e893b8c11b","Type":"ContainerStarted","Data":"736f2a74171e9a02078a116c0dafed5825972d8b72c356908210ddf748108428"} Nov 03 14:25:28.378101 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:28.377697 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7322-a13e07" event={"ID":"10f26a66-e9a1-4412-9362-b3e893b8c11b","Type":"ContainerStarted","Data":"9716457158a0a000b1c37aec9b3647e17e8807fbb38ad65986f51539d2292f8d"} Nov 03 14:25:28.416732 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:28.416680 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7322-a13e07" podStartSLOduration=1.41666629 podStartE2EDuration="1.41666629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:25:28.414413612 +0000 UTC m=+862.303203193" watchObservedRunningTime="2026-11-03 14:25:28.41666629 +0000 UTC m=+862.305455871" Nov 03 14:25:29.284720 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:29.284684 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7322-a13e07"] Nov 03 14:25:29.290788 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:29.290760 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7322-a13e07"] Nov 03 14:25:29.446315 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:29.446269 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7322-b87bb1"] Nov 03 14:25:29.446804 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:29.446736 5060 cpu_manager.go:414] "containerMap: removing container" podUID="10f26a66-e9a1-4412-9362-b3e893b8c11b" containerName="test-curl-7322-a13e07" Nov 03 14:25:29.446804 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:29.446750 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10f26a66-e9a1-4412-9362-b3e893b8c11b" containerName="test-curl-7322-a13e07" Nov 03 14:25:29.446947 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:29.446859 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="10f26a66-e9a1-4412-9362-b3e893b8c11b" containerName="test-curl-7322-a13e07" Nov 03 14:25:29.449698 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:29.449677 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-b87bb1" Nov 03 14:25:29.472560 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:29.472535 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7322-b87bb1"] Nov 03 14:25:29.550243 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:29.550155 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmppm\" (UniqueName: \"kubernetes.io/projected/688ae80d-938f-4eff-90cc-6844e1a94e72-kube-api-access-qmppm\") pod \"test-curl-7322-b87bb1\" (UID: \"688ae80d-938f-4eff-90cc-6844e1a94e72\") " pod="opendatahub/test-curl-7322-b87bb1" Nov 03 14:25:29.651618 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:29.651565 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmppm\" (UniqueName: \"kubernetes.io/projected/688ae80d-938f-4eff-90cc-6844e1a94e72-kube-api-access-qmppm\") pod \"test-curl-7322-b87bb1\" (UID: \"688ae80d-938f-4eff-90cc-6844e1a94e72\") " pod="opendatahub/test-curl-7322-b87bb1" Nov 03 14:25:29.671296 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:29.671268 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmppm\" (UniqueName: \"kubernetes.io/projected/688ae80d-938f-4eff-90cc-6844e1a94e72-kube-api-access-qmppm\") pod \"test-curl-7322-b87bb1\" (UID: \"688ae80d-938f-4eff-90cc-6844e1a94e72\") " pod="opendatahub/test-curl-7322-b87bb1" Nov 03 14:25:29.762285 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:29.762249 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-b87bb1" Nov 03 14:25:30.074952 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:30.074925 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7322-b87bb1"] Nov 03 14:25:30.076657 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:25:30.076626 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod688ae80d_938f_4eff_90cc_6844e1a94e72.slice/crio-10b169a5ce1fd15e17748a74b518f99e1d7c324199a98e099ac136f44345f051 WatchSource:0}: Error finding container 10b169a5ce1fd15e17748a74b518f99e1d7c324199a98e099ac136f44345f051: Status 404 returned error can't find the container with id 10b169a5ce1fd15e17748a74b518f99e1d7c324199a98e099ac136f44345f051 Nov 03 14:25:30.392552 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:30.392458 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7322-b87bb1" event={"ID":"688ae80d-938f-4eff-90cc-6844e1a94e72","Type":"ContainerStarted","Data":"8385ec8124cae52b63fc983aa466b852ebbdfa4e1b46d18ff1e64f88bdd7bfbb"} Nov 03 14:25:30.392552 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:30.392508 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7322-b87bb1" event={"ID":"688ae80d-938f-4eff-90cc-6844e1a94e72","Type":"ContainerStarted","Data":"10b169a5ce1fd15e17748a74b518f99e1d7c324199a98e099ac136f44345f051"} Nov 03 14:25:30.392784 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:30.392724 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7322-a13e07" podUID="10f26a66-e9a1-4412-9362-b3e893b8c11b" containerName="test-curl-7322-a13e07" containerID="cri-o://736f2a74171e9a02078a116c0dafed5825972d8b72c356908210ddf748108428" gracePeriod=2 Nov 03 14:25:30.431661 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:30.431610 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7322-b87bb1" podStartSLOduration=1.431595685 podStartE2EDuration="1.431595685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:25:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:25:30.430465183 +0000 UTC m=+864.319254767" watchObservedRunningTime="2026-11-03 14:25:30.431595685 +0000 UTC m=+864.320385300" Nov 03 14:25:31.375862 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:31.369536 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7322-b87bb1"] Nov 03 14:25:31.381056 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:31.381019 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7322-b87bb1"] Nov 03 14:25:31.538122 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:31.538084 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4"] Nov 03 14:25:31.539130 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:31.539099 5060 cpu_manager.go:414] "containerMap: removing container" podUID="688ae80d-938f-4eff-90cc-6844e1a94e72" containerName="test-curl-7322-b87bb1" Nov 03 14:25:31.539312 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:31.539295 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="688ae80d-938f-4eff-90cc-6844e1a94e72" containerName="test-curl-7322-b87bb1" Nov 03 14:25:31.539597 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:31.539582 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="688ae80d-938f-4eff-90cc-6844e1a94e72" containerName="test-curl-7322-b87bb1" Nov 03 14:25:31.548583 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:31.548549 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4"] Nov 03 14:25:31.548821 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:31.548810 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7322-03bdd6"] Nov 03 14:25:31.556562 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:31.555422 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" Nov 03 14:25:31.568893 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:31.567665 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-03bdd6" Nov 03 14:25:31.595857 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:31.583119 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-b4361f-openshift-default-dockercfg-h8cr5\"" Nov 03 14:25:31.724150 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:31.724116 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7322-03bdd6"] Nov 03 14:25:31.821975 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:31.821916 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d073cee6-107f-4a7f-8117-f48de055e723-workload-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4\" (UID: \"d073cee6-107f-4a7f-8117-f48de055e723\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" Nov 03 14:25:31.822185 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:31.821991 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnsql\" (UniqueName: \"kubernetes.io/projected/e54c5e51-8df7-4fbc-a772-b51c2e20d4e5-kube-api-access-jnsql\") pod \"test-curl-7322-03bdd6\" (UID: \"e54c5e51-8df7-4fbc-a772-b51c2e20d4e5\") " pod="opendatahub/test-curl-7322-03bdd6" Nov 03 14:25:31.822185 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:31.822092 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d073cee6-107f-4a7f-8117-f48de055e723-istio-envoy\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4\" (UID: \"d073cee6-107f-4a7f-8117-f48de055e723\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" Nov 03 14:25:31.822185 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:31.822143 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d073cee6-107f-4a7f-8117-f48de055e723-credential-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4\" (UID: \"d073cee6-107f-4a7f-8117-f48de055e723\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" Nov 03 14:25:31.822350 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:31.822210 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d073cee6-107f-4a7f-8117-f48de055e723-istio-data\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4\" (UID: \"d073cee6-107f-4a7f-8117-f48de055e723\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" Nov 03 14:25:31.822350 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:31.822277 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d073cee6-107f-4a7f-8117-f48de055e723-istio-podinfo\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4\" (UID: \"d073cee6-107f-4a7f-8117-f48de055e723\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" Nov 03 14:25:31.822350 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:31.822301 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d073cee6-107f-4a7f-8117-f48de055e723-istio-token\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4\" (UID: \"d073cee6-107f-4a7f-8117-f48de055e723\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" Nov 03 14:25:31.822350 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:31.822332 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ql9q\" (UniqueName: \"kubernetes.io/projected/d073cee6-107f-4a7f-8117-f48de055e723-kube-api-access-5ql9q\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4\" (UID: \"d073cee6-107f-4a7f-8117-f48de055e723\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" Nov 03 14:25:31.822541 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:31.822387 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d073cee6-107f-4a7f-8117-f48de055e723-istiod-ca-cert\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4\" (UID: \"d073cee6-107f-4a7f-8117-f48de055e723\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" Nov 03 14:25:31.822541 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:31.822420 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d073cee6-107f-4a7f-8117-f48de055e723-workload-certs\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4\" (UID: \"d073cee6-107f-4a7f-8117-f48de055e723\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" Nov 03 14:25:31.923423 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:31.923380 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnsql\" (UniqueName: \"kubernetes.io/projected/e54c5e51-8df7-4fbc-a772-b51c2e20d4e5-kube-api-access-jnsql\") pod \"test-curl-7322-03bdd6\" (UID: \"e54c5e51-8df7-4fbc-a772-b51c2e20d4e5\") " pod="opendatahub/test-curl-7322-03bdd6" Nov 03 14:25:31.923632 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:31.923448 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d073cee6-107f-4a7f-8117-f48de055e723-istio-envoy\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4\" (UID: \"d073cee6-107f-4a7f-8117-f48de055e723\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" Nov 03 14:25:31.923632 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:31.923483 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d073cee6-107f-4a7f-8117-f48de055e723-credential-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4\" (UID: \"d073cee6-107f-4a7f-8117-f48de055e723\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" Nov 03 14:25:31.923632 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:31.923534 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d073cee6-107f-4a7f-8117-f48de055e723-istio-data\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4\" (UID: \"d073cee6-107f-4a7f-8117-f48de055e723\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" Nov 03 14:25:31.923632 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:31.923600 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d073cee6-107f-4a7f-8117-f48de055e723-istio-podinfo\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4\" (UID: \"d073cee6-107f-4a7f-8117-f48de055e723\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" Nov 03 14:25:31.923632 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:31.923626 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d073cee6-107f-4a7f-8117-f48de055e723-istio-token\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4\" (UID: \"d073cee6-107f-4a7f-8117-f48de055e723\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" Nov 03 14:25:31.923913 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:31.923656 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ql9q\" (UniqueName: \"kubernetes.io/projected/d073cee6-107f-4a7f-8117-f48de055e723-kube-api-access-5ql9q\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4\" (UID: \"d073cee6-107f-4a7f-8117-f48de055e723\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" Nov 03 14:25:31.923913 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:31.923715 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d073cee6-107f-4a7f-8117-f48de055e723-istiod-ca-cert\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4\" (UID: \"d073cee6-107f-4a7f-8117-f48de055e723\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" Nov 03 14:25:31.923913 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:31.923749 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d073cee6-107f-4a7f-8117-f48de055e723-workload-certs\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4\" (UID: \"d073cee6-107f-4a7f-8117-f48de055e723\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" Nov 03 14:25:31.923913 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:31.923900 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d073cee6-107f-4a7f-8117-f48de055e723-workload-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4\" (UID: \"d073cee6-107f-4a7f-8117-f48de055e723\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" Nov 03 14:25:31.924116 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:31.924072 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d073cee6-107f-4a7f-8117-f48de055e723-credential-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4\" (UID: \"d073cee6-107f-4a7f-8117-f48de055e723\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" Nov 03 14:25:31.924284 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:31.924254 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d073cee6-107f-4a7f-8117-f48de055e723-istio-data\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4\" (UID: \"d073cee6-107f-4a7f-8117-f48de055e723\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" Nov 03 14:25:31.924423 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:31.924264 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d073cee6-107f-4a7f-8117-f48de055e723-workload-certs\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4\" (UID: \"d073cee6-107f-4a7f-8117-f48de055e723\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" Nov 03 14:25:31.924423 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:31.924355 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d073cee6-107f-4a7f-8117-f48de055e723-workload-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4\" (UID: \"d073cee6-107f-4a7f-8117-f48de055e723\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" Nov 03 14:25:31.924927 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:31.924902 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d073cee6-107f-4a7f-8117-f48de055e723-istiod-ca-cert\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4\" (UID: \"d073cee6-107f-4a7f-8117-f48de055e723\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" Nov 03 14:25:31.926662 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:31.926642 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d073cee6-107f-4a7f-8117-f48de055e723-istio-podinfo\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4\" (UID: \"d073cee6-107f-4a7f-8117-f48de055e723\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" Nov 03 14:25:31.928336 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:31.928314 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d073cee6-107f-4a7f-8117-f48de055e723-istio-envoy\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4\" (UID: \"d073cee6-107f-4a7f-8117-f48de055e723\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" Nov 03 14:25:31.951723 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:31.951606 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d073cee6-107f-4a7f-8117-f48de055e723-istio-token\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4\" (UID: \"d073cee6-107f-4a7f-8117-f48de055e723\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" Nov 03 14:25:31.952013 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:31.951984 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnsql\" (UniqueName: \"kubernetes.io/projected/e54c5e51-8df7-4fbc-a772-b51c2e20d4e5-kube-api-access-jnsql\") pod \"test-curl-7322-03bdd6\" (UID: \"e54c5e51-8df7-4fbc-a772-b51c2e20d4e5\") " pod="opendatahub/test-curl-7322-03bdd6" Nov 03 14:25:31.962679 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:31.962646 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ql9q\" (UniqueName: \"kubernetes.io/projected/d073cee6-107f-4a7f-8117-f48de055e723-kube-api-access-5ql9q\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4\" (UID: \"d073cee6-107f-4a7f-8117-f48de055e723\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" Nov 03 14:25:32.046101 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:32.046066 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" Nov 03 14:25:32.074357 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:32.074324 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-03bdd6" Nov 03 14:25:32.497826 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:32.463742 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7322-b87bb1" podUID="688ae80d-938f-4eff-90cc-6844e1a94e72" containerName="test-curl-7322-b87bb1" containerID="cri-o://8385ec8124cae52b63fc983aa466b852ebbdfa4e1b46d18ff1e64f88bdd7bfbb" gracePeriod=2 Nov 03 14:25:33.036874 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:33.015134 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4"] Nov 03 14:25:33.237873 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:33.226096 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:25:33.237873 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:33.226248 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:25:33.237873 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:33.226342 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:25:33.342109 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:33.335053 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7322-03bdd6"] Nov 03 14:25:33.478566 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:33.478445 5060 generic.go:356] "Generic (PLEG): container finished" podID="10f26a66-e9a1-4412-9362-b3e893b8c11b" containerID="736f2a74171e9a02078a116c0dafed5825972d8b72c356908210ddf748108428" exitCode=137 Nov 03 14:25:33.480581 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:33.480496 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" event={"ID":"d073cee6-107f-4a7f-8117-f48de055e723","Type":"ContainerStarted","Data":"c824f12ba8115293aa70ac08fb835a423cd38b6fadb7c80228938eca97faac56"} Nov 03 14:25:33.482297 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:33.482244 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7322-03bdd6" event={"ID":"e54c5e51-8df7-4fbc-a772-b51c2e20d4e5","Type":"ContainerStarted","Data":"7739722d961b8f1015683481a71d1a53fba4b27d5f3c10a0bff2f3306ade5758"} Nov 03 14:25:33.676574 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:33.676547 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-a13e07" Nov 03 14:25:33.683995 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:33.683954 5060 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:25:33.684117 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:33.684011 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" podUID="b3327f34-66f3-4a70-915a-aa5ad8685fad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:25:33.802858 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:33.802464 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10f26a66-e9a1-4412-9362-b3e893b8c11b-kube-api-access-xm74x\" (UniqueName: \"kubernetes.io/projected/10f26a66-e9a1-4412-9362-b3e893b8c11b-kube-api-access-xm74x\") pod \"10f26a66-e9a1-4412-9362-b3e893b8c11b\" (UID: \"10f26a66-e9a1-4412-9362-b3e893b8c11b\") " Nov 03 14:25:33.806855 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:33.805344 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f26a66-e9a1-4412-9362-b3e893b8c11b-kube-api-access-xm74x" pod "10f26a66-e9a1-4412-9362-b3e893b8c11b" (UID: "10f26a66-e9a1-4412-9362-b3e893b8c11b"). InnerVolumeSpecName "kube-api-access-xm74x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:25:33.903917 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:33.903876 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm74x\" (UniqueName: \"kubernetes.io/projected/10f26a66-e9a1-4412-9362-b3e893b8c11b-kube-api-access-xm74x\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:25:34.250344 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:34.250311 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10f26a66-e9a1-4412-9362-b3e893b8c11b" path="/var/lib/kubelet/pods/10f26a66-e9a1-4412-9362-b3e893b8c11b/volumes" Nov 03 14:25:34.490089 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:34.490061 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-a13e07" Nov 03 14:25:34.490089 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:34.490078 5060 scope.go:122] "RemoveContainer" containerID="736f2a74171e9a02078a116c0dafed5825972d8b72c356908210ddf748108428" Nov 03 14:25:34.492643 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:34.491925 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" event={"ID":"d073cee6-107f-4a7f-8117-f48de055e723","Type":"ContainerStarted","Data":"3a2d2a056be578877ce5697216806387880856c9389bcbb2718b214e15251ab6"} Nov 03 14:25:34.493891 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:34.493864 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7322-03bdd6" event={"ID":"e54c5e51-8df7-4fbc-a772-b51c2e20d4e5","Type":"ContainerStarted","Data":"c0f7189ac34cc0a5add5075a6546b6bdb8cd4d0fb8c13cafab0dcb1bdb9e1901"} Nov 03 14:25:34.557152 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:34.557100 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" podStartSLOduration=3.557087187 podStartE2EDuration="3.557087187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:25:34.544156794 +0000 UTC m=+868.432946469" watchObservedRunningTime="2026-11-03 14:25:34.557087187 +0000 UTC m=+868.445876768" Nov 03 14:25:34.585625 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:34.585576 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7322-03bdd6" podStartSLOduration=3.585562512 podStartE2EDuration="3.585562512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:25:34.582985497 +0000 UTC m=+868.471775078" watchObservedRunningTime="2026-11-03 14:25:34.585562512 +0000 UTC m=+868.474352092" Nov 03 14:25:34.933150 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:34.933117 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-b87bb1" Nov 03 14:25:35.018693 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:35.018655 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/688ae80d-938f-4eff-90cc-6844e1a94e72-kube-api-access-qmppm\" (UniqueName: \"kubernetes.io/projected/688ae80d-938f-4eff-90cc-6844e1a94e72-kube-api-access-qmppm\") pod \"688ae80d-938f-4eff-90cc-6844e1a94e72\" (UID: \"688ae80d-938f-4eff-90cc-6844e1a94e72\") " Nov 03 14:25:35.021008 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:35.020978 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/688ae80d-938f-4eff-90cc-6844e1a94e72-kube-api-access-qmppm" pod "688ae80d-938f-4eff-90cc-6844e1a94e72" (UID: "688ae80d-938f-4eff-90cc-6844e1a94e72"). InnerVolumeSpecName "kube-api-access-qmppm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:25:35.047211 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:35.047135 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" Nov 03 14:25:35.052358 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:35.052336 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" Nov 03 14:25:35.120083 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:35.120041 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmppm\" (UniqueName: \"kubernetes.io/projected/688ae80d-938f-4eff-90cc-6844e1a94e72-kube-api-access-qmppm\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:25:35.427761 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:35.427667 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7322-03bdd6"] Nov 03 14:25:35.431897 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:35.431867 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7322-03bdd6"] Nov 03 14:25:35.502101 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:35.502064 5060 generic.go:356] "Generic (PLEG): container finished" podID="688ae80d-938f-4eff-90cc-6844e1a94e72" containerID="8385ec8124cae52b63fc983aa466b852ebbdfa4e1b46d18ff1e64f88bdd7bfbb" exitCode=137 Nov 03 14:25:35.502273 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:35.502113 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-b87bb1" Nov 03 14:25:35.502273 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:35.502160 5060 scope.go:122] "RemoveContainer" containerID="8385ec8124cae52b63fc983aa466b852ebbdfa4e1b46d18ff1e64f88bdd7bfbb" Nov 03 14:25:35.504822 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:35.504773 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" Nov 03 14:25:35.505730 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:35.505708 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" Nov 03 14:25:35.513291 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:35.513265 5060 scope.go:122] "RemoveContainer" containerID="8385ec8124cae52b63fc983aa466b852ebbdfa4e1b46d18ff1e64f88bdd7bfbb" Nov 03 14:25:35.513595 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:25:35.513575 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8385ec8124cae52b63fc983aa466b852ebbdfa4e1b46d18ff1e64f88bdd7bfbb\": container with ID starting with 8385ec8124cae52b63fc983aa466b852ebbdfa4e1b46d18ff1e64f88bdd7bfbb not found: ID does not exist" containerID="8385ec8124cae52b63fc983aa466b852ebbdfa4e1b46d18ff1e64f88bdd7bfbb" Nov 03 14:25:35.513638 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:35.513604 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8385ec8124cae52b63fc983aa466b852ebbdfa4e1b46d18ff1e64f88bdd7bfbb"} err="failed to get container status \"8385ec8124cae52b63fc983aa466b852ebbdfa4e1b46d18ff1e64f88bdd7bfbb\": rpc error: code = NotFound desc = could not find container \"8385ec8124cae52b63fc983aa466b852ebbdfa4e1b46d18ff1e64f88bdd7bfbb\": container with ID starting with 8385ec8124cae52b63fc983aa466b852ebbdfa4e1b46d18ff1e64f88bdd7bfbb not found: ID does not exist" Nov 03 14:25:35.558993 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:25:35.558948 5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7322-b87bb1\" is forbidden: User \"system:node:ip-10-0-143-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-171.ec2.internal' and this object" podUID="688ae80d-938f-4eff-90cc-6844e1a94e72" pod="opendatahub/test-curl-7322-b87bb1" Nov 03 14:25:35.582585 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:35.582546 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7322-843fbc"] Nov 03 14:25:35.583588 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:35.583558 5060 cpu_manager.go:414] "containerMap: removing container" podUID="e54c5e51-8df7-4fbc-a772-b51c2e20d4e5" containerName="test-curl-7322-03bdd6" Nov 03 14:25:35.583747 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:35.583731 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e54c5e51-8df7-4fbc-a772-b51c2e20d4e5" containerName="test-curl-7322-03bdd6" Nov 03 14:25:35.583984 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:35.583966 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e54c5e51-8df7-4fbc-a772-b51c2e20d4e5" containerName="test-curl-7322-03bdd6" Nov 03 14:25:35.589226 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:35.589206 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-843fbc" Nov 03 14:25:35.601706 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:25:35.601675 5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7322-b87bb1\" is forbidden: User \"system:node:ip-10-0-143-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-171.ec2.internal' and this object" podUID="688ae80d-938f-4eff-90cc-6844e1a94e72" pod="opendatahub/test-curl-7322-b87bb1" Nov 03 14:25:35.612722 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:35.612693 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7322-843fbc"] Nov 03 14:25:35.727375 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:35.727286 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kqwt\" (UniqueName: \"kubernetes.io/projected/45cc65a2-83c5-43d0-b369-076635b05eec-kube-api-access-5kqwt\") pod \"test-curl-7322-843fbc\" (UID: \"45cc65a2-83c5-43d0-b369-076635b05eec\") " pod="opendatahub/test-curl-7322-843fbc" Nov 03 14:25:35.828924 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:35.828888 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kqwt\" (UniqueName: \"kubernetes.io/projected/45cc65a2-83c5-43d0-b369-076635b05eec-kube-api-access-5kqwt\") pod \"test-curl-7322-843fbc\" (UID: \"45cc65a2-83c5-43d0-b369-076635b05eec\") " pod="opendatahub/test-curl-7322-843fbc" Nov 03 14:25:35.849064 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:35.849031 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kqwt\" (UniqueName: \"kubernetes.io/projected/45cc65a2-83c5-43d0-b369-076635b05eec-kube-api-access-5kqwt\") pod \"test-curl-7322-843fbc\" (UID: \"45cc65a2-83c5-43d0-b369-076635b05eec\") " pod="opendatahub/test-curl-7322-843fbc" Nov 03 14:25:35.905124 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:35.905096 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-843fbc" Nov 03 14:25:36.211008 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:36.210975 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7322-843fbc"] Nov 03 14:25:36.212592 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:25:36.212558 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45cc65a2_83c5_43d0_b369_076635b05eec.slice/crio-e338fce9862f57ffe204423147ec580ead8c93c1ad9ad852337e973129ba4e63 WatchSource:0}: Error finding container e338fce9862f57ffe204423147ec580ead8c93c1ad9ad852337e973129ba4e63: Status 404 returned error can't find the container with id e338fce9862f57ffe204423147ec580ead8c93c1ad9ad852337e973129ba4e63 Nov 03 14:25:36.249913 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:36.249881 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="688ae80d-938f-4eff-90cc-6844e1a94e72" path="/var/lib/kubelet/pods/688ae80d-938f-4eff-90cc-6844e1a94e72/volumes" Nov 03 14:25:36.258547 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:25:36.258511 5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7322-b87bb1\" is forbidden: User \"system:node:ip-10-0-143-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-171.ec2.internal' and this object" podUID="688ae80d-938f-4eff-90cc-6844e1a94e72" pod="opendatahub/test-curl-7322-b87bb1" Nov 03 14:25:36.514415 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:36.514376 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7322-843fbc" event={"ID":"45cc65a2-83c5-43d0-b369-076635b05eec","Type":"ContainerStarted","Data":"929207b3b32a3ccf06eb105e57876dbd6c0e74f2d145ae66bb9b0099744b4635"} Nov 03 14:25:36.514600 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:36.514431 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7322-843fbc" event={"ID":"45cc65a2-83c5-43d0-b369-076635b05eec","Type":"ContainerStarted","Data":"e338fce9862f57ffe204423147ec580ead8c93c1ad9ad852337e973129ba4e63"} Nov 03 14:25:36.515829 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:36.515790 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7322-03bdd6" podUID="e54c5e51-8df7-4fbc-a772-b51c2e20d4e5" containerName="test-curl-7322-03bdd6" containerID="cri-o://c0f7189ac34cc0a5add5075a6546b6bdb8cd4d0fb8c13cafab0dcb1bdb9e1901" gracePeriod=2 Nov 03 14:25:36.553073 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:36.553024 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7322-843fbc" podStartSLOduration=1.553010716 podStartE2EDuration="1.553010716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:25:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:25:36.550505808 +0000 UTC m=+870.439295400" watchObservedRunningTime="2026-11-03 14:25:36.553010716 +0000 UTC m=+870.441800296" Nov 03 14:25:37.469175 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:37.469136 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7322-843fbc"] Nov 03 14:25:37.476043 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:37.475999 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7322-843fbc"] Nov 03 14:25:38.531641 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:38.531601 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7322-843fbc" podUID="45cc65a2-83c5-43d0-b369-076635b05eec" containerName="test-curl-7322-843fbc" containerID="cri-o://929207b3b32a3ccf06eb105e57876dbd6c0e74f2d145ae66bb9b0099744b4635" gracePeriod=2 Nov 03 14:25:38.736917 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:38.736891 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-03bdd6" Nov 03 14:25:38.865502 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:38.865410 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e54c5e51-8df7-4fbc-a772-b51c2e20d4e5-kube-api-access-jnsql\" (UniqueName: \"kubernetes.io/projected/e54c5e51-8df7-4fbc-a772-b51c2e20d4e5-kube-api-access-jnsql\") pod \"e54c5e51-8df7-4fbc-a772-b51c2e20d4e5\" (UID: \"e54c5e51-8df7-4fbc-a772-b51c2e20d4e5\") " Nov 03 14:25:38.867537 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:38.867509 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e54c5e51-8df7-4fbc-a772-b51c2e20d4e5-kube-api-access-jnsql" pod "e54c5e51-8df7-4fbc-a772-b51c2e20d4e5" (UID: "e54c5e51-8df7-4fbc-a772-b51c2e20d4e5"). InnerVolumeSpecName "kube-api-access-jnsql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:25:38.967023 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:38.966985 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnsql\" (UniqueName: \"kubernetes.io/projected/e54c5e51-8df7-4fbc-a772-b51c2e20d4e5-kube-api-access-jnsql\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:25:39.440959 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:25:39.440919 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:25:39.539027 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:39.538991 5060 generic.go:356] "Generic (PLEG): container finished" podID="e54c5e51-8df7-4fbc-a772-b51c2e20d4e5" containerID="c0f7189ac34cc0a5add5075a6546b6bdb8cd4d0fb8c13cafab0dcb1bdb9e1901" exitCode=137 Nov 03 14:25:39.539502 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:39.539039 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-03bdd6" Nov 03 14:25:39.539502 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:39.539078 5060 scope.go:122] "RemoveContainer" containerID="c0f7189ac34cc0a5add5075a6546b6bdb8cd4d0fb8c13cafab0dcb1bdb9e1901" Nov 03 14:25:39.549999 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:39.549976 5060 scope.go:122] "RemoveContainer" containerID="c0f7189ac34cc0a5add5075a6546b6bdb8cd4d0fb8c13cafab0dcb1bdb9e1901" Nov 03 14:25:39.550290 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:25:39.550269 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0f7189ac34cc0a5add5075a6546b6bdb8cd4d0fb8c13cafab0dcb1bdb9e1901\": container with ID starting with c0f7189ac34cc0a5add5075a6546b6bdb8cd4d0fb8c13cafab0dcb1bdb9e1901 not found: ID does not exist" containerID="c0f7189ac34cc0a5add5075a6546b6bdb8cd4d0fb8c13cafab0dcb1bdb9e1901" Nov 03 14:25:39.550365 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:39.550299 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0f7189ac34cc0a5add5075a6546b6bdb8cd4d0fb8c13cafab0dcb1bdb9e1901"} err="failed to get container status \"c0f7189ac34cc0a5add5075a6546b6bdb8cd4d0fb8c13cafab0dcb1bdb9e1901\": rpc error: code = NotFound desc = could not find container \"c0f7189ac34cc0a5add5075a6546b6bdb8cd4d0fb8c13cafab0dcb1bdb9e1901\": container with ID starting with c0f7189ac34cc0a5add5075a6546b6bdb8cd4d0fb8c13cafab0dcb1bdb9e1901 not found: ID does not exist" Nov 03 14:25:39.561380 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:25:39.561346 5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7322-03bdd6\" is forbidden: User \"system:node:ip-10-0-143-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-171.ec2.internal' and this object" podUID="e54c5e51-8df7-4fbc-a772-b51c2e20d4e5" pod="opendatahub/test-curl-7322-03bdd6" Nov 03 14:25:40.250674 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:40.250637 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e54c5e51-8df7-4fbc-a772-b51c2e20d4e5" path="/var/lib/kubelet/pods/e54c5e51-8df7-4fbc-a772-b51c2e20d4e5/volumes" Nov 03 14:25:40.743177 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:40.743143 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-843fbc" Nov 03 14:25:40.898441 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:40.898346 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45cc65a2-83c5-43d0-b369-076635b05eec-kube-api-access-5kqwt\" (UniqueName: \"kubernetes.io/projected/45cc65a2-83c5-43d0-b369-076635b05eec-kube-api-access-5kqwt\") pod \"45cc65a2-83c5-43d0-b369-076635b05eec\" (UID: \"45cc65a2-83c5-43d0-b369-076635b05eec\") " Nov 03 14:25:40.922815 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:40.920881 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45cc65a2-83c5-43d0-b369-076635b05eec-kube-api-access-5kqwt" pod "45cc65a2-83c5-43d0-b369-076635b05eec" (UID: "45cc65a2-83c5-43d0-b369-076635b05eec"). InnerVolumeSpecName "kube-api-access-5kqwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:25:40.999697 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:40.999654 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kqwt\" (UniqueName: \"kubernetes.io/projected/45cc65a2-83c5-43d0-b369-076635b05eec-kube-api-access-5kqwt\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:25:41.556006 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:41.555968 5060 generic.go:356] "Generic (PLEG): container finished" podID="45cc65a2-83c5-43d0-b369-076635b05eec" containerID="929207b3b32a3ccf06eb105e57876dbd6c0e74f2d145ae66bb9b0099744b4635" exitCode=137 Nov 03 14:25:41.556205 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:41.556031 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-843fbc" Nov 03 14:25:41.556205 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:41.556058 5060 scope.go:122] "RemoveContainer" containerID="929207b3b32a3ccf06eb105e57876dbd6c0e74f2d145ae66bb9b0099744b4635" Nov 03 14:25:41.565564 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:41.565427 5060 scope.go:122] "RemoveContainer" containerID="929207b3b32a3ccf06eb105e57876dbd6c0e74f2d145ae66bb9b0099744b4635" Nov 03 14:25:41.565748 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:25:41.565727 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"929207b3b32a3ccf06eb105e57876dbd6c0e74f2d145ae66bb9b0099744b4635\": container with ID starting with 929207b3b32a3ccf06eb105e57876dbd6c0e74f2d145ae66bb9b0099744b4635 not found: ID does not exist" containerID="929207b3b32a3ccf06eb105e57876dbd6c0e74f2d145ae66bb9b0099744b4635" Nov 03 14:25:41.565791 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:41.565756 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"929207b3b32a3ccf06eb105e57876dbd6c0e74f2d145ae66bb9b0099744b4635"} err="failed to get container status \"929207b3b32a3ccf06eb105e57876dbd6c0e74f2d145ae66bb9b0099744b4635\": rpc error: code = NotFound desc = could not find container \"929207b3b32a3ccf06eb105e57876dbd6c0e74f2d145ae66bb9b0099744b4635\": container with ID starting with 929207b3b32a3ccf06eb105e57876dbd6c0e74f2d145ae66bb9b0099744b4635 not found: ID does not exist" Nov 03 14:25:41.576756 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:25:41.576727 5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7322-843fbc\" is forbidden: User \"system:node:ip-10-0-143-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-171.ec2.internal' and this object" podUID="45cc65a2-83c5-43d0-b369-076635b05eec" pod="opendatahub/test-curl-7322-843fbc" Nov 03 14:25:42.252510 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:42.252474 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45cc65a2-83c5-43d0-b369-076635b05eec" path="/var/lib/kubelet/pods/45cc65a2-83c5-43d0-b369-076635b05eec/volumes" Nov 03 14:25:42.768148 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:42.768111 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-1ffaa0/rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz"] Nov 03 14:25:42.777027 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:42.768923 5060 cpu_manager.go:414] "containerMap: removing container" podUID="45cc65a2-83c5-43d0-b369-076635b05eec" containerName="test-curl-7322-843fbc" Nov 03 14:25:42.777027 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:42.768947 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45cc65a2-83c5-43d0-b369-076635b05eec" containerName="test-curl-7322-843fbc" Nov 03 14:25:42.777027 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:42.769117 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="45cc65a2-83c5-43d0-b369-076635b05eec" containerName="test-curl-7322-843fbc" Nov 03 14:25:42.777027 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:42.774522 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz" Nov 03 14:25:42.808787 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:42.808752 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-1ffaa0\"/\"openshift-service-ca.crt\"" Nov 03 14:25:42.809141 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:42.809127 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-1ffaa0\"/\"default-dockercfg-gk8b4\"" Nov 03 14:25:42.809274 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:42.809186 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-1ffaa0/rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz"] Nov 03 14:25:42.809386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:42.809310 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-1ffaa0\"/\"kube-root-ca.crt\"" Nov 03 14:25:42.809574 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:42.809407 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-1ffaa0\"/\"rate-test-model-1ffaa0-kserve-self-signed-certs\"" Nov 03 14:25:42.927078 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:42.926440 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/930bcc45-2ffe-4502-aff6-03761f25ee62-tls-certs\") pod \"rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz\" (UID: \"930bcc45-2ffe-4502-aff6-03761f25ee62\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz" Nov 03 14:25:42.927078 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:42.926506 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/930bcc45-2ffe-4502-aff6-03761f25ee62-home\") pod \"rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz\" (UID: \"930bcc45-2ffe-4502-aff6-03761f25ee62\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz" Nov 03 14:25:42.927078 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:42.926566 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/930bcc45-2ffe-4502-aff6-03761f25ee62-tmp-dir\") pod \"rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz\" (UID: \"930bcc45-2ffe-4502-aff6-03761f25ee62\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz" Nov 03 14:25:42.927078 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:42.926731 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/930bcc45-2ffe-4502-aff6-03761f25ee62-dshm\") pod \"rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz\" (UID: \"930bcc45-2ffe-4502-aff6-03761f25ee62\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz" Nov 03 14:25:42.927078 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:42.926810 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/930bcc45-2ffe-4502-aff6-03761f25ee62-model-cache\") pod \"rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz\" (UID: \"930bcc45-2ffe-4502-aff6-03761f25ee62\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz" Nov 03 14:25:42.927078 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:42.926884 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v77lx\" (UniqueName: \"kubernetes.io/projected/930bcc45-2ffe-4502-aff6-03761f25ee62-kube-api-access-v77lx\") pod \"rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz\" (UID: \"930bcc45-2ffe-4502-aff6-03761f25ee62\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz" Nov 03 14:25:43.030916 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:43.030491 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/930bcc45-2ffe-4502-aff6-03761f25ee62-tmp-dir\") pod \"rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz\" (UID: \"930bcc45-2ffe-4502-aff6-03761f25ee62\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz" Nov 03 14:25:43.030916 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:43.030604 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/930bcc45-2ffe-4502-aff6-03761f25ee62-dshm\") pod \"rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz\" (UID: \"930bcc45-2ffe-4502-aff6-03761f25ee62\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz" Nov 03 14:25:43.030916 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:43.030672 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/930bcc45-2ffe-4502-aff6-03761f25ee62-model-cache\") pod \"rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz\" (UID: \"930bcc45-2ffe-4502-aff6-03761f25ee62\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz" Nov 03 14:25:43.030916 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:43.030711 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v77lx\" (UniqueName: \"kubernetes.io/projected/930bcc45-2ffe-4502-aff6-03761f25ee62-kube-api-access-v77lx\") pod \"rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz\" (UID: \"930bcc45-2ffe-4502-aff6-03761f25ee62\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz" Nov 03 14:25:43.030916 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:43.030858 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/930bcc45-2ffe-4502-aff6-03761f25ee62-tls-certs\") pod \"rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz\" (UID: \"930bcc45-2ffe-4502-aff6-03761f25ee62\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz" Nov 03 14:25:43.030916 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:43.030903 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/930bcc45-2ffe-4502-aff6-03761f25ee62-home\") pod \"rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz\" (UID: \"930bcc45-2ffe-4502-aff6-03761f25ee62\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz" Nov 03 14:25:43.036024 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:43.031363 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/930bcc45-2ffe-4502-aff6-03761f25ee62-home\") pod \"rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz\" (UID: \"930bcc45-2ffe-4502-aff6-03761f25ee62\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz" Nov 03 14:25:43.036024 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:43.031617 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/930bcc45-2ffe-4502-aff6-03761f25ee62-tmp-dir\") pod \"rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz\" (UID: \"930bcc45-2ffe-4502-aff6-03761f25ee62\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz" Nov 03 14:25:43.036024 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:43.032566 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/930bcc45-2ffe-4502-aff6-03761f25ee62-model-cache\") pod \"rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz\" (UID: \"930bcc45-2ffe-4502-aff6-03761f25ee62\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz" Nov 03 14:25:43.069861 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:43.069618 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v77lx\" (UniqueName: \"kubernetes.io/projected/930bcc45-2ffe-4502-aff6-03761f25ee62-kube-api-access-v77lx\") pod \"rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz\" (UID: \"930bcc45-2ffe-4502-aff6-03761f25ee62\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz" Nov 03 14:25:43.080888 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:43.075469 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/930bcc45-2ffe-4502-aff6-03761f25ee62-tls-certs\") pod \"rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz\" (UID: \"930bcc45-2ffe-4502-aff6-03761f25ee62\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz" Nov 03 14:25:43.100926 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:43.084125 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/930bcc45-2ffe-4502-aff6-03761f25ee62-dshm\") pod \"rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz\" (UID: \"930bcc45-2ffe-4502-aff6-03761f25ee62\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz" Nov 03 14:25:43.123862 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:43.117223 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz" Nov 03 14:25:43.656049 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:43.643442 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-1ffaa0/rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz"] Nov 03 14:25:44.672861 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:44.672066 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz" event={"ID":"930bcc45-2ffe-4502-aff6-03761f25ee62","Type":"ContainerStarted","Data":"6039b6e672816c985ab88f05f2940ccdf4de8db764d78ce2a15d546a528abd64"} Nov 03 14:25:44.672861 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:44.672122 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz" event={"ID":"930bcc45-2ffe-4502-aff6-03761f25ee62","Type":"ContainerStarted","Data":"7c2a51a397fdba01b2f9e0aaa02cfa26c14537fc7f7e6e54833ff12f712fa94d"} Nov 03 14:25:45.126858 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:45.120935 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz" Nov 03 14:25:48.087921 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:48.087864 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz" podStartSLOduration=6.087850231 podStartE2EDuration="6.087850231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:25:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:25:44.722688843 +0000 UTC m=+878.611478427" watchObservedRunningTime="2026-11-03 14:25:48.087850231 +0000 UTC m=+881.976639802" Nov 03 14:25:48.090573 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:48.090539 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81c16996-v4vdl"] Nov 03 14:25:48.094693 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:48.094671 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c16996-v4vdl" Nov 03 14:25:48.110063 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:48.110034 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:25:48.116106 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:48.115701 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dkhm\" (UniqueName: \"kubernetes.io/projected/bf486f2f-1712-40b4-8af2-e8e5a9f8aaae-kube-api-access-7dkhm\") pod \"maas-api-revoke-sub-81c16996-v4vdl\" (UID: \"bf486f2f-1712-40b4-8af2-e8e5a9f8aaae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c16996-v4vdl" Nov 03 14:25:48.116106 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:48.115778 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf486f2f-1712-40b4-8af2-e8e5a9f8aaae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81c16996-v4vdl\" (UID: \"bf486f2f-1712-40b4-8af2-e8e5a9f8aaae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c16996-v4vdl" Nov 03 14:25:48.116106 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:48.115827 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf486f2f-1712-40b4-8af2-e8e5a9f8aaae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81c16996-v4vdl\" (UID: \"bf486f2f-1712-40b4-8af2-e8e5a9f8aaae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c16996-v4vdl" Nov 03 14:25:48.121577 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:48.121551 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:25:48.121719 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:48.121556 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\"" Nov 03 14:25:48.121719 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:48.121561 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:25:48.127539 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:48.127492 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81c16996-v4vdl"] Nov 03 14:25:48.217022 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:48.216989 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dkhm\" (UniqueName: \"kubernetes.io/projected/bf486f2f-1712-40b4-8af2-e8e5a9f8aaae-kube-api-access-7dkhm\") pod \"maas-api-revoke-sub-81c16996-v4vdl\" (UID: \"bf486f2f-1712-40b4-8af2-e8e5a9f8aaae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c16996-v4vdl" Nov 03 14:25:48.217206 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:48.217043 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf486f2f-1712-40b4-8af2-e8e5a9f8aaae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81c16996-v4vdl\" (UID: \"bf486f2f-1712-40b4-8af2-e8e5a9f8aaae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c16996-v4vdl" Nov 03 14:25:48.217206 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:48.217183 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf486f2f-1712-40b4-8af2-e8e5a9f8aaae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81c16996-v4vdl\" (UID: \"bf486f2f-1712-40b4-8af2-e8e5a9f8aaae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c16996-v4vdl" Nov 03 14:25:48.217782 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:48.217760 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf486f2f-1712-40b4-8af2-e8e5a9f8aaae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81c16996-v4vdl\" (UID: \"bf486f2f-1712-40b4-8af2-e8e5a9f8aaae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c16996-v4vdl" Nov 03 14:25:48.219698 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:48.219668 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf486f2f-1712-40b4-8af2-e8e5a9f8aaae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81c16996-v4vdl\" (UID: \"bf486f2f-1712-40b4-8af2-e8e5a9f8aaae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c16996-v4vdl" Nov 03 14:25:48.238498 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:48.238476 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dkhm\" (UniqueName: \"kubernetes.io/projected/bf486f2f-1712-40b4-8af2-e8e5a9f8aaae-kube-api-access-7dkhm\") pod \"maas-api-revoke-sub-81c16996-v4vdl\" (UID: \"bf486f2f-1712-40b4-8af2-e8e5a9f8aaae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c16996-v4vdl" Nov 03 14:25:48.408552 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:48.408467 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c16996-v4vdl" Nov 03 14:25:48.721897 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:48.721864 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81c16996-v4vdl"] Nov 03 14:25:48.723377 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:25:48.723347 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf486f2f_1712_40b4_8af2_e8e5a9f8aaae.slice/crio-209d6a864437a59423cbe5dd8dfdc125c47f4738868af7026095b35d8cfe53fe WatchSource:0}: Error finding container 209d6a864437a59423cbe5dd8dfdc125c47f4738868af7026095b35d8cfe53fe: Status 404 returned error can't find the container with id 209d6a864437a59423cbe5dd8dfdc125c47f4738868af7026095b35d8cfe53fe Nov 03 14:25:49.709156 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:49.709115 5060 generic.go:356] "Generic (PLEG): container finished" podID="bf486f2f-1712-40b4-8af2-e8e5a9f8aaae" containerID="d228f7837b3cbe360b8d4af736e9ef3b60f7ded190cd0fd547f4c7d6dea560a0" exitCode=0 Nov 03 14:25:49.709570 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:49.709208 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c16996-v4vdl" event={"ID":"bf486f2f-1712-40b4-8af2-e8e5a9f8aaae","Type":"ContainerDied","Data":"d228f7837b3cbe360b8d4af736e9ef3b60f7ded190cd0fd547f4c7d6dea560a0"} Nov 03 14:25:49.709570 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:49.709274 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c16996-v4vdl" event={"ID":"bf486f2f-1712-40b4-8af2-e8e5a9f8aaae","Type":"ContainerStarted","Data":"209d6a864437a59423cbe5dd8dfdc125c47f4738868af7026095b35d8cfe53fe"} Nov 03 14:25:50.960828 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:50.960799 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c16996-v4vdl" Nov 03 14:25:50.998702 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:50.998674 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf486f2f-1712-40b4-8af2-e8e5a9f8aaae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf486f2f-1712-40b4-8af2-e8e5a9f8aaae-maas-api-openshift-ca\") pod \"bf486f2f-1712-40b4-8af2-e8e5a9f8aaae\" (UID: \"bf486f2f-1712-40b4-8af2-e8e5a9f8aaae\") " Nov 03 14:25:50.998917 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:50.998730 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf486f2f-1712-40b4-8af2-e8e5a9f8aaae-kube-api-access-7dkhm\" (UniqueName: \"kubernetes.io/projected/bf486f2f-1712-40b4-8af2-e8e5a9f8aaae-kube-api-access-7dkhm\") pod \"bf486f2f-1712-40b4-8af2-e8e5a9f8aaae\" (UID: \"bf486f2f-1712-40b4-8af2-e8e5a9f8aaae\") " Nov 03 14:25:50.998995 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:50.998970 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf486f2f-1712-40b4-8af2-e8e5a9f8aaae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf486f2f-1712-40b4-8af2-e8e5a9f8aaae-maas-api-xks-ca\") pod \"bf486f2f-1712-40b4-8af2-e8e5a9f8aaae\" (UID: \"bf486f2f-1712-40b4-8af2-e8e5a9f8aaae\") " Nov 03 14:25:50.999318 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:50.999203 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf486f2f-1712-40b4-8af2-e8e5a9f8aaae-maas-api-openshift-ca" pod "bf486f2f-1712-40b4-8af2-e8e5a9f8aaae" (UID: "bf486f2f-1712-40b4-8af2-e8e5a9f8aaae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:25:50.999587 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:50.999565 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf486f2f-1712-40b4-8af2-e8e5a9f8aaae-maas-api-openshift-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:25:51.011901 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:51.001520 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf486f2f-1712-40b4-8af2-e8e5a9f8aaae-kube-api-access-7dkhm" pod "bf486f2f-1712-40b4-8af2-e8e5a9f8aaae" (UID: "bf486f2f-1712-40b4-8af2-e8e5a9f8aaae"). InnerVolumeSpecName "kube-api-access-7dkhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:25:51.011901 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:51.001828 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf486f2f-1712-40b4-8af2-e8e5a9f8aaae-maas-api-xks-ca" pod "bf486f2f-1712-40b4-8af2-e8e5a9f8aaae" (UID: "bf486f2f-1712-40b4-8af2-e8e5a9f8aaae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:25:51.100751 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:51.100712 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf486f2f-1712-40b4-8af2-e8e5a9f8aaae-maas-api-xks-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:25:51.100751 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:51.100748 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dkhm\" (UniqueName: \"kubernetes.io/projected/bf486f2f-1712-40b4-8af2-e8e5a9f8aaae-kube-api-access-7dkhm\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:25:51.758310 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:51.758268 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c16996-v4vdl" event={"ID":"bf486f2f-1712-40b4-8af2-e8e5a9f8aaae","Type":"ContainerDied","Data":"209d6a864437a59423cbe5dd8dfdc125c47f4738868af7026095b35d8cfe53fe"} Nov 03 14:25:51.758561 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:51.758542 5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="209d6a864437a59423cbe5dd8dfdc125c47f4738868af7026095b35d8cfe53fe" Nov 03 14:25:51.758738 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:51.758724 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c16996-v4vdl" Nov 03 14:25:53.118201 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:53.118169 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz" Nov 03 14:25:53.131286 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:53.131253 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz" Nov 03 14:25:53.781542 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:25:53.781516 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz" Nov 03 14:26:06.382238 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:06.382206 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/3.log" Nov 03 14:26:06.385282 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:06.385256 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/3.log" Nov 03 14:26:09.450210 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:26:09.450165 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:26:12.083344 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:12.083265 5060 generic.go:356] "Generic (PLEG): container finished" podID="d073cee6-107f-4a7f-8117-f48de055e723" containerID="3a2d2a056be578877ce5697216806387880856c9389bcbb2718b214e15251ab6" exitCode=0 Nov 03 14:26:12.083779 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:12.083340 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" event={"ID":"d073cee6-107f-4a7f-8117-f48de055e723","Type":"ContainerDied","Data":"3a2d2a056be578877ce5697216806387880856c9389bcbb2718b214e15251ab6"} Nov 03 14:26:12.083779 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:12.083732 5060 scope.go:122] "RemoveContainer" containerID="3a2d2a056be578877ce5697216806387880856c9389bcbb2718b214e15251ab6" Nov 03 14:26:12.085258 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:12.085224 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:26:12.085369 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:12.085311 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:26:12.085369 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:12.085358 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:26:13.092177 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:13.092144 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" event={"ID":"d073cee6-107f-4a7f-8117-f48de055e723","Type":"ContainerStarted","Data":"9fca417cc6621f31361585bf48ee6da880ad1cb71a7c8a62e2112b4cb6f61e41"} Nov 03 14:26:14.046638 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:14.046607 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" Nov 03 14:26:14.046954 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:14.046922 5060 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 14:26:14.047035 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:14.046995 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" podUID="d073cee6-107f-4a7f-8117-f48de055e723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 14:26:15.047035 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:15.047000 5060 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 14:26:15.047447 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:15.047056 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" podUID="d073cee6-107f-4a7f-8117-f48de055e723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 14:26:16.047826 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:16.047774 5060 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 14:26:16.048353 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:16.047851 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" podUID="d073cee6-107f-4a7f-8117-f48de055e723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 14:26:17.047487 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:17.047448 5060 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 14:26:17.047671 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:17.047567 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" podUID="d073cee6-107f-4a7f-8117-f48de055e723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 14:26:18.047858 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:18.047165 5060 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 14:26:18.047858 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:18.047243 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" podUID="d073cee6-107f-4a7f-8117-f48de055e723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 14:26:19.051747 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:19.051712 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" Nov 03 14:26:19.052150 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:19.052129 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" Nov 03 14:26:19.052739 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:19.052717 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" Nov 03 14:26:25.345809 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:25.345770 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55be1c47-tqqbg"] Nov 03 14:26:25.346328 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:25.346313 5060 cpu_manager.go:414] "containerMap: removing container" podUID="bf486f2f-1712-40b4-8af2-e8e5a9f8aaae" containerName="revoke-keys" Nov 03 14:26:25.346372 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:25.346329 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf486f2f-1712-40b4-8af2-e8e5a9f8aaae" containerName="revoke-keys" Nov 03 14:26:25.346427 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:25.346418 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf486f2f-1712-40b4-8af2-e8e5a9f8aaae" containerName="revoke-keys" Nov 03 14:26:25.349737 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:25.349721 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55be1c47-tqqbg" Nov 03 14:26:25.380099 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:25.380070 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:26:25.380310 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:25.380072 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:26:25.380410 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:25.380184 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\"" Nov 03 14:26:25.380484 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:25.380319 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:26:25.384083 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:25.384056 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55be1c47-tqqbg"] Nov 03 14:26:25.497013 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:25.496982 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/209a19fc-8c5d-406a-9b4d-32a4e9ff49f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55be1c47-tqqbg\" (UID: \"209a19fc-8c5d-406a-9b4d-32a4e9ff49f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55be1c47-tqqbg" Nov 03 14:26:25.497201 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:25.497057 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/209a19fc-8c5d-406a-9b4d-32a4e9ff49f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55be1c47-tqqbg\" (UID: \"209a19fc-8c5d-406a-9b4d-32a4e9ff49f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55be1c47-tqqbg" Nov 03 14:26:25.497201 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:25.497155 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7mjd\" (UniqueName: \"kubernetes.io/projected/209a19fc-8c5d-406a-9b4d-32a4e9ff49f3-kube-api-access-k7mjd\") pod \"maas-api-revoke-sub-55be1c47-tqqbg\" (UID: \"209a19fc-8c5d-406a-9b4d-32a4e9ff49f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55be1c47-tqqbg" Nov 03 14:26:25.598483 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:25.598389 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7mjd\" (UniqueName: \"kubernetes.io/projected/209a19fc-8c5d-406a-9b4d-32a4e9ff49f3-kube-api-access-k7mjd\") pod \"maas-api-revoke-sub-55be1c47-tqqbg\" (UID: \"209a19fc-8c5d-406a-9b4d-32a4e9ff49f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55be1c47-tqqbg" Nov 03 14:26:25.598659 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:25.598554 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/209a19fc-8c5d-406a-9b4d-32a4e9ff49f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55be1c47-tqqbg\" (UID: \"209a19fc-8c5d-406a-9b4d-32a4e9ff49f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55be1c47-tqqbg" Nov 03 14:26:25.598659 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:25.598626 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/209a19fc-8c5d-406a-9b4d-32a4e9ff49f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55be1c47-tqqbg\" (UID: \"209a19fc-8c5d-406a-9b4d-32a4e9ff49f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55be1c47-tqqbg" Nov 03 14:26:25.599310 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:25.599288 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/209a19fc-8c5d-406a-9b4d-32a4e9ff49f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55be1c47-tqqbg\" (UID: \"209a19fc-8c5d-406a-9b4d-32a4e9ff49f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55be1c47-tqqbg" Nov 03 14:26:25.600938 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:25.600920 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/209a19fc-8c5d-406a-9b4d-32a4e9ff49f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55be1c47-tqqbg\" (UID: \"209a19fc-8c5d-406a-9b4d-32a4e9ff49f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55be1c47-tqqbg" Nov 03 14:26:25.620402 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:25.620378 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7mjd\" (UniqueName: \"kubernetes.io/projected/209a19fc-8c5d-406a-9b4d-32a4e9ff49f3-kube-api-access-k7mjd\") pod \"maas-api-revoke-sub-55be1c47-tqqbg\" (UID: \"209a19fc-8c5d-406a-9b4d-32a4e9ff49f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55be1c47-tqqbg" Nov 03 14:26:25.663912 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:25.663879 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55be1c47-tqqbg" Nov 03 14:26:25.976431 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:25.976318 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55be1c47-tqqbg"] Nov 03 14:26:25.981183 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:26:25.981156 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod209a19fc_8c5d_406a_9b4d_32a4e9ff49f3.slice/crio-5f6b2c99c833102307c66a2b869f37ec663db1f5b6dac989c7d34c8a5b62775d WatchSource:0}: Error finding container 5f6b2c99c833102307c66a2b869f37ec663db1f5b6dac989c7d34c8a5b62775d: Status 404 returned error can't find the container with id 5f6b2c99c833102307c66a2b869f37ec663db1f5b6dac989c7d34c8a5b62775d Nov 03 14:26:26.221513 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:26.221474 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55be1c47-tqqbg" event={"ID":"209a19fc-8c5d-406a-9b4d-32a4e9ff49f3","Type":"ContainerStarted","Data":"5f6b2c99c833102307c66a2b869f37ec663db1f5b6dac989c7d34c8a5b62775d"} Nov 03 14:26:27.230401 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:27.230309 5060 generic.go:356] "Generic (PLEG): container finished" podID="209a19fc-8c5d-406a-9b4d-32a4e9ff49f3" containerID="80b31063a03c108d34d4a251b774e1cf03d4a8e58f562a1dce5f256007e3b14c" exitCode=0 Nov 03 14:26:27.230813 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:27.230393 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55be1c47-tqqbg" event={"ID":"209a19fc-8c5d-406a-9b4d-32a4e9ff49f3","Type":"ContainerDied","Data":"80b31063a03c108d34d4a251b774e1cf03d4a8e58f562a1dce5f256007e3b14c"} Nov 03 14:26:28.398757 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:28.398721 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55be1c47-tqqbg" Nov 03 14:26:28.532643 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:28.532606 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/209a19fc-8c5d-406a-9b4d-32a4e9ff49f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/209a19fc-8c5d-406a-9b4d-32a4e9ff49f3-maas-api-xks-ca\") pod \"209a19fc-8c5d-406a-9b4d-32a4e9ff49f3\" (UID: \"209a19fc-8c5d-406a-9b4d-32a4e9ff49f3\") " Nov 03 14:26:28.532859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:28.532719 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/209a19fc-8c5d-406a-9b4d-32a4e9ff49f3-kube-api-access-k7mjd\" (UniqueName: \"kubernetes.io/projected/209a19fc-8c5d-406a-9b4d-32a4e9ff49f3-kube-api-access-k7mjd\") pod \"209a19fc-8c5d-406a-9b4d-32a4e9ff49f3\" (UID: \"209a19fc-8c5d-406a-9b4d-32a4e9ff49f3\") " Nov 03 14:26:28.532859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:28.532784 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/209a19fc-8c5d-406a-9b4d-32a4e9ff49f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/209a19fc-8c5d-406a-9b4d-32a4e9ff49f3-maas-api-openshift-ca\") pod \"209a19fc-8c5d-406a-9b4d-32a4e9ff49f3\" (UID: \"209a19fc-8c5d-406a-9b4d-32a4e9ff49f3\") " Nov 03 14:26:28.533228 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:28.533196 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/209a19fc-8c5d-406a-9b4d-32a4e9ff49f3-maas-api-openshift-ca" pod "209a19fc-8c5d-406a-9b4d-32a4e9ff49f3" (UID: "209a19fc-8c5d-406a-9b4d-32a4e9ff49f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:26:28.535316 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:28.535281 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/209a19fc-8c5d-406a-9b4d-32a4e9ff49f3-maas-api-xks-ca" pod "209a19fc-8c5d-406a-9b4d-32a4e9ff49f3" (UID: "209a19fc-8c5d-406a-9b4d-32a4e9ff49f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:26:28.535485 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:28.535461 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/209a19fc-8c5d-406a-9b4d-32a4e9ff49f3-kube-api-access-k7mjd" pod "209a19fc-8c5d-406a-9b4d-32a4e9ff49f3" (UID: "209a19fc-8c5d-406a-9b4d-32a4e9ff49f3"). InnerVolumeSpecName "kube-api-access-k7mjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:26:28.633777 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:28.633745 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7mjd\" (UniqueName: \"kubernetes.io/projected/209a19fc-8c5d-406a-9b4d-32a4e9ff49f3-kube-api-access-k7mjd\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:26:28.633777 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:28.633776 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/209a19fc-8c5d-406a-9b4d-32a4e9ff49f3-maas-api-openshift-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:26:28.633996 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:28.633789 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/209a19fc-8c5d-406a-9b4d-32a4e9ff49f3-maas-api-xks-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:26:29.244763 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:29.244732 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55be1c47-tqqbg" Nov 03 14:26:29.244965 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:29.244766 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55be1c47-tqqbg" event={"ID":"209a19fc-8c5d-406a-9b4d-32a4e9ff49f3","Type":"ContainerDied","Data":"5f6b2c99c833102307c66a2b869f37ec663db1f5b6dac989c7d34c8a5b62775d"} Nov 03 14:26:29.244965 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:29.244820 5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f6b2c99c833102307c66a2b869f37ec663db1f5b6dac989c7d34c8a5b62775d" Nov 03 14:26:39.443762 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:26:39.443722 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:26:40.012904 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:40.012868 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9454069-48k8g"] Nov 03 14:26:40.013386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:40.013370 5060 cpu_manager.go:414] "containerMap: removing container" podUID="209a19fc-8c5d-406a-9b4d-32a4e9ff49f3" containerName="revoke-keys" Nov 03 14:26:40.013386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:40.013386 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="209a19fc-8c5d-406a-9b4d-32a4e9ff49f3" containerName="revoke-keys" Nov 03 14:26:40.013540 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:40.013482 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="209a19fc-8c5d-406a-9b4d-32a4e9ff49f3" containerName="revoke-keys" Nov 03 14:26:40.017170 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:40.017142 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9454069-48k8g" Nov 03 14:26:40.033361 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:40.033342 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\"" Nov 03 14:26:40.043815 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:40.043793 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:26:40.043990 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:40.043968 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:26:40.044048 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:40.043977 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:26:40.047690 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:40.047665 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9454069-48k8g"] Nov 03 14:26:40.132495 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:40.132455 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23aa4bb8-5671-497f-9792-4bba9a67ed61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9454069-48k8g\" (UID: \"23aa4bb8-5671-497f-9792-4bba9a67ed61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9454069-48k8g" Nov 03 14:26:40.132698 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:40.132532 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpqf7\" (UniqueName: \"kubernetes.io/projected/23aa4bb8-5671-497f-9792-4bba9a67ed61-kube-api-access-qpqf7\") pod \"maas-api-revoke-sub-e9454069-48k8g\" (UID: \"23aa4bb8-5671-497f-9792-4bba9a67ed61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9454069-48k8g" Nov 03 14:26:40.132698 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:40.132637 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23aa4bb8-5671-497f-9792-4bba9a67ed61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9454069-48k8g\" (UID: \"23aa4bb8-5671-497f-9792-4bba9a67ed61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9454069-48k8g" Nov 03 14:26:40.234023 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:40.233974 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpqf7\" (UniqueName: \"kubernetes.io/projected/23aa4bb8-5671-497f-9792-4bba9a67ed61-kube-api-access-qpqf7\") pod \"maas-api-revoke-sub-e9454069-48k8g\" (UID: \"23aa4bb8-5671-497f-9792-4bba9a67ed61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9454069-48k8g" Nov 03 14:26:40.234023 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:40.234022 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23aa4bb8-5671-497f-9792-4bba9a67ed61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9454069-48k8g\" (UID: \"23aa4bb8-5671-497f-9792-4bba9a67ed61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9454069-48k8g" Nov 03 14:26:40.234281 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:40.234135 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23aa4bb8-5671-497f-9792-4bba9a67ed61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9454069-48k8g\" (UID: \"23aa4bb8-5671-497f-9792-4bba9a67ed61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9454069-48k8g" Nov 03 14:26:40.234788 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:40.234764 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23aa4bb8-5671-497f-9792-4bba9a67ed61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9454069-48k8g\" (UID: \"23aa4bb8-5671-497f-9792-4bba9a67ed61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9454069-48k8g" Nov 03 14:26:40.236544 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:40.236515 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23aa4bb8-5671-497f-9792-4bba9a67ed61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9454069-48k8g\" (UID: \"23aa4bb8-5671-497f-9792-4bba9a67ed61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9454069-48k8g" Nov 03 14:26:40.254792 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:40.254765 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpqf7\" (UniqueName: \"kubernetes.io/projected/23aa4bb8-5671-497f-9792-4bba9a67ed61-kube-api-access-qpqf7\") pod \"maas-api-revoke-sub-e9454069-48k8g\" (UID: \"23aa4bb8-5671-497f-9792-4bba9a67ed61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9454069-48k8g" Nov 03 14:26:40.331164 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:40.331069 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9454069-48k8g" Nov 03 14:26:40.641363 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:40.641332 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9454069-48k8g"] Nov 03 14:26:40.642534 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:26:40.642506 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23aa4bb8_5671_497f_9792_4bba9a67ed61.slice/crio-1efb60127f0ec0949b6df46dbc9b0927114d9b4e9e59e1182ab1c16d09384638 WatchSource:0}: Error finding container 1efb60127f0ec0949b6df46dbc9b0927114d9b4e9e59e1182ab1c16d09384638: Status 404 returned error can't find the container with id 1efb60127f0ec0949b6df46dbc9b0927114d9b4e9e59e1182ab1c16d09384638 Nov 03 14:26:41.411900 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:41.411864 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9454069-48k8g" event={"ID":"23aa4bb8-5671-497f-9792-4bba9a67ed61","Type":"ContainerStarted","Data":"9ea5a9be6931d78841bd7f963c782b594996607eff6ed6f65a70532210c29779"} Nov 03 14:26:41.412079 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:41.411919 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9454069-48k8g" event={"ID":"23aa4bb8-5671-497f-9792-4bba9a67ed61","Type":"ContainerStarted","Data":"1efb60127f0ec0949b6df46dbc9b0927114d9b4e9e59e1182ab1c16d09384638"} Nov 03 14:26:41.456663 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:41.456619 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9454069-48k8g" podStartSLOduration=2.4566052640000002 podStartE2EDuration="2.456605264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:26:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:26:41.455355221 +0000 UTC m=+935.344144802" watchObservedRunningTime="2026-11-03 14:26:41.456605264 +0000 UTC m=+935.345394844" Nov 03 14:26:42.420369 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:42.420333 5060 generic.go:356] "Generic (PLEG): container finished" podID="23aa4bb8-5671-497f-9792-4bba9a67ed61" containerID="9ea5a9be6931d78841bd7f963c782b594996607eff6ed6f65a70532210c29779" exitCode=0 Nov 03 14:26:42.420811 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:42.420422 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9454069-48k8g" event={"ID":"23aa4bb8-5671-497f-9792-4bba9a67ed61","Type":"ContainerDied","Data":"9ea5a9be6931d78841bd7f963c782b594996607eff6ed6f65a70532210c29779"} Nov 03 14:26:44.493053 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:44.493017 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9454069-48k8g" event={"ID":"23aa4bb8-5671-497f-9792-4bba9a67ed61","Type":"ContainerDied","Data":"1efb60127f0ec0949b6df46dbc9b0927114d9b4e9e59e1182ab1c16d09384638"} Nov 03 14:26:44.493698 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:44.493656 5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1efb60127f0ec0949b6df46dbc9b0927114d9b4e9e59e1182ab1c16d09384638" Nov 03 14:26:44.570869 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:44.562714 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9454069-48k8g" Nov 03 14:26:44.655859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:44.655063 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23aa4bb8-5671-497f-9792-4bba9a67ed61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23aa4bb8-5671-497f-9792-4bba9a67ed61-maas-api-openshift-ca\") pod \"23aa4bb8-5671-497f-9792-4bba9a67ed61\" (UID: \"23aa4bb8-5671-497f-9792-4bba9a67ed61\") " Nov 03 14:26:44.655859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:44.655183 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23aa4bb8-5671-497f-9792-4bba9a67ed61-kube-api-access-qpqf7\" (UniqueName: \"kubernetes.io/projected/23aa4bb8-5671-497f-9792-4bba9a67ed61-kube-api-access-qpqf7\") pod \"23aa4bb8-5671-497f-9792-4bba9a67ed61\" (UID: \"23aa4bb8-5671-497f-9792-4bba9a67ed61\") " Nov 03 14:26:44.655859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:44.655280 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23aa4bb8-5671-497f-9792-4bba9a67ed61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23aa4bb8-5671-497f-9792-4bba9a67ed61-maas-api-xks-ca\") pod \"23aa4bb8-5671-497f-9792-4bba9a67ed61\" (UID: \"23aa4bb8-5671-497f-9792-4bba9a67ed61\") " Nov 03 14:26:44.660860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:44.656432 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23aa4bb8-5671-497f-9792-4bba9a67ed61-maas-api-openshift-ca" pod "23aa4bb8-5671-497f-9792-4bba9a67ed61" (UID: "23aa4bb8-5671-497f-9792-4bba9a67ed61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:26:44.684133 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:44.680298 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23aa4bb8-5671-497f-9792-4bba9a67ed61-kube-api-access-qpqf7" pod "23aa4bb8-5671-497f-9792-4bba9a67ed61" (UID: "23aa4bb8-5671-497f-9792-4bba9a67ed61"). InnerVolumeSpecName "kube-api-access-qpqf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:26:44.684133 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:44.684042 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23aa4bb8-5671-497f-9792-4bba9a67ed61-maas-api-xks-ca" pod "23aa4bb8-5671-497f-9792-4bba9a67ed61" (UID: "23aa4bb8-5671-497f-9792-4bba9a67ed61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:26:44.761701 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:44.761660 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23aa4bb8-5671-497f-9792-4bba9a67ed61-maas-api-openshift-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:26:44.761954 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:44.761933 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpqf7\" (UniqueName: \"kubernetes.io/projected/23aa4bb8-5671-497f-9792-4bba9a67ed61-kube-api-access-qpqf7\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:26:44.762087 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:44.762073 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23aa4bb8-5671-497f-9792-4bba9a67ed61-maas-api-xks-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:26:45.502164 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:45.502125 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9454069-48k8g" Nov 03 14:26:53.146940 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:53.146860 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf1bc777-cc6qj"] Nov 03 14:26:53.147814 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:53.147786 5060 cpu_manager.go:414] "containerMap: removing container" podUID="23aa4bb8-5671-497f-9792-4bba9a67ed61" containerName="revoke-keys" Nov 03 14:26:53.147814 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:53.147813 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23aa4bb8-5671-497f-9792-4bba9a67ed61" containerName="revoke-keys" Nov 03 14:26:53.147985 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:53.147975 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="23aa4bb8-5671-497f-9792-4bba9a67ed61" containerName="revoke-keys" Nov 03 14:26:53.151501 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:53.151475 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf1bc777-cc6qj" Nov 03 14:26:53.178127 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:53.178098 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf1bc777-cc6qj"] Nov 03 14:26:53.179115 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:53.179088 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0229f45e-0f1c-42dd-b0bc-27b15a88392b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf1bc777-cc6qj\" (UID: \"0229f45e-0f1c-42dd-b0bc-27b15a88392b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf1bc777-cc6qj" Nov 03 14:26:53.179209 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:53.179149 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0229f45e-0f1c-42dd-b0bc-27b15a88392b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf1bc777-cc6qj\" (UID: \"0229f45e-0f1c-42dd-b0bc-27b15a88392b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf1bc777-cc6qj" Nov 03 14:26:53.179257 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:53.179229 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6nhj\" (UniqueName: \"kubernetes.io/projected/0229f45e-0f1c-42dd-b0bc-27b15a88392b-kube-api-access-m6nhj\") pod \"maas-api-revoke-sub-cf1bc777-cc6qj\" (UID: \"0229f45e-0f1c-42dd-b0bc-27b15a88392b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf1bc777-cc6qj" Nov 03 14:26:53.184230 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:53.184205 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\"" Nov 03 14:26:53.184374 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:53.184270 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:26:53.184374 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:53.184317 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:26:53.184374 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:53.184335 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:26:53.280074 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:53.280038 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0229f45e-0f1c-42dd-b0bc-27b15a88392b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf1bc777-cc6qj\" (UID: \"0229f45e-0f1c-42dd-b0bc-27b15a88392b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf1bc777-cc6qj" Nov 03 14:26:53.280274 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:53.280089 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0229f45e-0f1c-42dd-b0bc-27b15a88392b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf1bc777-cc6qj\" (UID: \"0229f45e-0f1c-42dd-b0bc-27b15a88392b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf1bc777-cc6qj" Nov 03 14:26:53.280274 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:53.280218 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6nhj\" (UniqueName: \"kubernetes.io/projected/0229f45e-0f1c-42dd-b0bc-27b15a88392b-kube-api-access-m6nhj\") pod \"maas-api-revoke-sub-cf1bc777-cc6qj\" (UID: \"0229f45e-0f1c-42dd-b0bc-27b15a88392b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf1bc777-cc6qj" Nov 03 14:26:53.280829 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:53.280803 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0229f45e-0f1c-42dd-b0bc-27b15a88392b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf1bc777-cc6qj\" (UID: \"0229f45e-0f1c-42dd-b0bc-27b15a88392b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf1bc777-cc6qj" Nov 03 14:26:53.282764 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:53.282740 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0229f45e-0f1c-42dd-b0bc-27b15a88392b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf1bc777-cc6qj\" (UID: \"0229f45e-0f1c-42dd-b0bc-27b15a88392b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf1bc777-cc6qj" Nov 03 14:26:53.302035 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:53.301996 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6nhj\" (UniqueName: \"kubernetes.io/projected/0229f45e-0f1c-42dd-b0bc-27b15a88392b-kube-api-access-m6nhj\") pod \"maas-api-revoke-sub-cf1bc777-cc6qj\" (UID: \"0229f45e-0f1c-42dd-b0bc-27b15a88392b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf1bc777-cc6qj" Nov 03 14:26:53.467677 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:53.467581 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf1bc777-cc6qj" Nov 03 14:26:53.788686 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:53.788656 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf1bc777-cc6qj"] Nov 03 14:26:53.789323 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:26:53.789300 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0229f45e_0f1c_42dd_b0bc_27b15a88392b.slice/crio-3e85960108e885a0af9b75f10c2fd5efaf23d8cbe4863a002b0ed852ecde71fd WatchSource:0}: Error finding container 3e85960108e885a0af9b75f10c2fd5efaf23d8cbe4863a002b0ed852ecde71fd: Status 404 returned error can't find the container with id 3e85960108e885a0af9b75f10c2fd5efaf23d8cbe4863a002b0ed852ecde71fd Nov 03 14:26:54.564448 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:54.564408 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf1bc777-cc6qj" event={"ID":"0229f45e-0f1c-42dd-b0bc-27b15a88392b","Type":"ContainerStarted","Data":"343bc6ba378af87be42a14a083e8fd195974426ab4ad1d1fa7e75913b795bf8e"} Nov 03 14:26:54.564883 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:54.564474 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf1bc777-cc6qj" event={"ID":"0229f45e-0f1c-42dd-b0bc-27b15a88392b","Type":"ContainerStarted","Data":"3e85960108e885a0af9b75f10c2fd5efaf23d8cbe4863a002b0ed852ecde71fd"} Nov 03 14:26:54.609991 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:54.609934 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf1bc777-cc6qj" podStartSLOduration=1.609919691 podStartE2EDuration="1.609919691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:26:54.608713323 +0000 UTC m=+948.497502907" watchObservedRunningTime="2026-11-03 14:26:54.609919691 +0000 UTC m=+948.498709263" Nov 03 14:26:55.573210 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:55.573173 5060 generic.go:356] "Generic (PLEG): container finished" podID="0229f45e-0f1c-42dd-b0bc-27b15a88392b" containerID="343bc6ba378af87be42a14a083e8fd195974426ab4ad1d1fa7e75913b795bf8e" exitCode=0 Nov 03 14:26:55.573210 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:55.573215 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf1bc777-cc6qj" event={"ID":"0229f45e-0f1c-42dd-b0bc-27b15a88392b","Type":"ContainerDied","Data":"343bc6ba378af87be42a14a083e8fd195974426ab4ad1d1fa7e75913b795bf8e"} Nov 03 14:26:57.084862 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:57.044047 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf1bc777-cc6qj" Nov 03 14:26:57.134871 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:57.120576 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0229f45e-0f1c-42dd-b0bc-27b15a88392b-kube-api-access-m6nhj\" (UniqueName: \"kubernetes.io/projected/0229f45e-0f1c-42dd-b0bc-27b15a88392b-kube-api-access-m6nhj\") pod \"0229f45e-0f1c-42dd-b0bc-27b15a88392b\" (UID: \"0229f45e-0f1c-42dd-b0bc-27b15a88392b\") " Nov 03 14:26:57.135401 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:57.135373 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0229f45e-0f1c-42dd-b0bc-27b15a88392b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0229f45e-0f1c-42dd-b0bc-27b15a88392b-maas-api-xks-ca\") pod \"0229f45e-0f1c-42dd-b0bc-27b15a88392b\" (UID: \"0229f45e-0f1c-42dd-b0bc-27b15a88392b\") " Nov 03 14:26:57.150042 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:57.142034 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0229f45e-0f1c-42dd-b0bc-27b15a88392b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0229f45e-0f1c-42dd-b0bc-27b15a88392b-maas-api-openshift-ca\") pod \"0229f45e-0f1c-42dd-b0bc-27b15a88392b\" (UID: \"0229f45e-0f1c-42dd-b0bc-27b15a88392b\") " Nov 03 14:26:57.150535 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:57.150484 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0229f45e-0f1c-42dd-b0bc-27b15a88392b-maas-api-openshift-ca" pod "0229f45e-0f1c-42dd-b0bc-27b15a88392b" (UID: "0229f45e-0f1c-42dd-b0bc-27b15a88392b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:26:57.193205 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:57.191978 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0229f45e-0f1c-42dd-b0bc-27b15a88392b-maas-api-xks-ca" pod "0229f45e-0f1c-42dd-b0bc-27b15a88392b" (UID: "0229f45e-0f1c-42dd-b0bc-27b15a88392b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:26:57.195928 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:57.195883 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0229f45e-0f1c-42dd-b0bc-27b15a88392b-kube-api-access-m6nhj" pod "0229f45e-0f1c-42dd-b0bc-27b15a88392b" (UID: "0229f45e-0f1c-42dd-b0bc-27b15a88392b"). InnerVolumeSpecName "kube-api-access-m6nhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:26:57.263860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:57.261472 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0229f45e-0f1c-42dd-b0bc-27b15a88392b-maas-api-xks-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:26:57.263860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:57.261513 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0229f45e-0f1c-42dd-b0bc-27b15a88392b-maas-api-openshift-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:26:57.263860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:57.261529 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6nhj\" (UniqueName: \"kubernetes.io/projected/0229f45e-0f1c-42dd-b0bc-27b15a88392b-kube-api-access-m6nhj\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:26:57.638857 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:57.635437 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf1bc777-cc6qj" event={"ID":"0229f45e-0f1c-42dd-b0bc-27b15a88392b","Type":"ContainerDied","Data":"3e85960108e885a0af9b75f10c2fd5efaf23d8cbe4863a002b0ed852ecde71fd"} Nov 03 14:26:57.638857 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:57.635505 5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e85960108e885a0af9b75f10c2fd5efaf23d8cbe4863a002b0ed852ecde71fd" Nov 03 14:26:57.638857 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:26:57.635582 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf1bc777-cc6qj" Nov 03 14:27:03.438630 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:03.438589 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9759f85a-lghlg"] Nov 03 14:27:03.439295 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:03.439272 5060 cpu_manager.go:414] "containerMap: removing container" podUID="0229f45e-0f1c-42dd-b0bc-27b15a88392b" containerName="revoke-keys" Nov 03 14:27:03.439400 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:03.439297 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0229f45e-0f1c-42dd-b0bc-27b15a88392b" containerName="revoke-keys" Nov 03 14:27:03.439481 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:03.439466 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="0229f45e-0f1c-42dd-b0bc-27b15a88392b" containerName="revoke-keys" Nov 03 14:27:03.443168 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:03.443146 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9759f85a-lghlg" Nov 03 14:27:03.471278 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:03.471251 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:27:03.471469 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:03.471273 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:27:03.471469 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:03.471271 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\"" Nov 03 14:27:03.471469 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:03.471289 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:27:03.476548 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:03.476518 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9759f85a-lghlg"] Nov 03 14:27:03.538178 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:03.538141 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blj95\" (UniqueName: \"kubernetes.io/projected/054d8392-289a-4eac-8926-ee85aaf15395-kube-api-access-blj95\") pod \"maas-api-revoke-sub-9759f85a-lghlg\" (UID: \"054d8392-289a-4eac-8926-ee85aaf15395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9759f85a-lghlg" Nov 03 14:27:03.538361 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:03.538206 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/054d8392-289a-4eac-8926-ee85aaf15395-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9759f85a-lghlg\" (UID: \"054d8392-289a-4eac-8926-ee85aaf15395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9759f85a-lghlg" Nov 03 14:27:03.538361 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:03.538304 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/054d8392-289a-4eac-8926-ee85aaf15395-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9759f85a-lghlg\" (UID: \"054d8392-289a-4eac-8926-ee85aaf15395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9759f85a-lghlg" Nov 03 14:27:03.640025 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:03.639990 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blj95\" (UniqueName: \"kubernetes.io/projected/054d8392-289a-4eac-8926-ee85aaf15395-kube-api-access-blj95\") pod \"maas-api-revoke-sub-9759f85a-lghlg\" (UID: \"054d8392-289a-4eac-8926-ee85aaf15395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9759f85a-lghlg" Nov 03 14:27:03.640230 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:03.640064 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/054d8392-289a-4eac-8926-ee85aaf15395-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9759f85a-lghlg\" (UID: \"054d8392-289a-4eac-8926-ee85aaf15395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9759f85a-lghlg" Nov 03 14:27:03.640230 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:03.640103 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/054d8392-289a-4eac-8926-ee85aaf15395-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9759f85a-lghlg\" (UID: \"054d8392-289a-4eac-8926-ee85aaf15395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9759f85a-lghlg" Nov 03 14:27:03.640731 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:03.640709 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/054d8392-289a-4eac-8926-ee85aaf15395-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9759f85a-lghlg\" (UID: \"054d8392-289a-4eac-8926-ee85aaf15395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9759f85a-lghlg" Nov 03 14:27:03.642507 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:03.642488 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/054d8392-289a-4eac-8926-ee85aaf15395-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9759f85a-lghlg\" (UID: \"054d8392-289a-4eac-8926-ee85aaf15395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9759f85a-lghlg" Nov 03 14:27:03.667728 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:03.667698 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blj95\" (UniqueName: \"kubernetes.io/projected/054d8392-289a-4eac-8926-ee85aaf15395-kube-api-access-blj95\") pod \"maas-api-revoke-sub-9759f85a-lghlg\" (UID: \"054d8392-289a-4eac-8926-ee85aaf15395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9759f85a-lghlg" Nov 03 14:27:03.756339 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:03.756303 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9759f85a-lghlg" Nov 03 14:27:04.071609 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:04.071579 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9759f85a-lghlg"] Nov 03 14:27:04.073334 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:27:04.073306 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod054d8392_289a_4eac_8926_ee85aaf15395.slice/crio-b9bd96da0c30db3a7c4da871720070a9b47465dd667962143a07458a8ce3ae4b WatchSource:0}: Error finding container b9bd96da0c30db3a7c4da871720070a9b47465dd667962143a07458a8ce3ae4b: Status 404 returned error can't find the container with id b9bd96da0c30db3a7c4da871720070a9b47465dd667962143a07458a8ce3ae4b Nov 03 14:27:04.685094 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:04.684995 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9759f85a-lghlg" event={"ID":"054d8392-289a-4eac-8926-ee85aaf15395","Type":"ContainerStarted","Data":"a879e953cb0939d131a77c634d70055dd83ac077134207efb3091b911f88bf05"} Nov 03 14:27:04.685094 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:04.685051 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9759f85a-lghlg" event={"ID":"054d8392-289a-4eac-8926-ee85aaf15395","Type":"ContainerStarted","Data":"b9bd96da0c30db3a7c4da871720070a9b47465dd667962143a07458a8ce3ae4b"} Nov 03 14:27:04.729361 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:04.729302 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9759f85a-lghlg" podStartSLOduration=1.729287934 podStartE2EDuration="1.729287934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:27:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:27:04.728023634 +0000 UTC m=+958.616813214" watchObservedRunningTime="2026-11-03 14:27:04.729287934 +0000 UTC m=+958.618077557" Nov 03 14:27:05.694818 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:05.694779 5060 generic.go:356] "Generic (PLEG): container finished" podID="054d8392-289a-4eac-8926-ee85aaf15395" containerID="a879e953cb0939d131a77c634d70055dd83ac077134207efb3091b911f88bf05" exitCode=0 Nov 03 14:27:05.695267 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:05.694867 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9759f85a-lghlg" event={"ID":"054d8392-289a-4eac-8926-ee85aaf15395","Type":"ContainerDied","Data":"a879e953cb0939d131a77c634d70055dd83ac077134207efb3091b911f88bf05"} Nov 03 14:27:06.868586 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:06.868558 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9759f85a-lghlg" Nov 03 14:27:06.981726 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:06.981650 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/054d8392-289a-4eac-8926-ee85aaf15395-kube-api-access-blj95\" (UniqueName: \"kubernetes.io/projected/054d8392-289a-4eac-8926-ee85aaf15395-kube-api-access-blj95\") pod \"054d8392-289a-4eac-8926-ee85aaf15395\" (UID: \"054d8392-289a-4eac-8926-ee85aaf15395\") " Nov 03 14:27:06.981906 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:06.981754 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/054d8392-289a-4eac-8926-ee85aaf15395-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/054d8392-289a-4eac-8926-ee85aaf15395-maas-api-xks-ca\") pod \"054d8392-289a-4eac-8926-ee85aaf15395\" (UID: \"054d8392-289a-4eac-8926-ee85aaf15395\") " Nov 03 14:27:06.981906 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:06.981845 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/054d8392-289a-4eac-8926-ee85aaf15395-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/054d8392-289a-4eac-8926-ee85aaf15395-maas-api-openshift-ca\") pod \"054d8392-289a-4eac-8926-ee85aaf15395\" (UID: \"054d8392-289a-4eac-8926-ee85aaf15395\") " Nov 03 14:27:06.982222 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:06.982194 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/054d8392-289a-4eac-8926-ee85aaf15395-maas-api-openshift-ca" pod "054d8392-289a-4eac-8926-ee85aaf15395" (UID: "054d8392-289a-4eac-8926-ee85aaf15395"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:27:06.982355 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:06.982301 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/054d8392-289a-4eac-8926-ee85aaf15395-maas-api-openshift-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:27:06.983751 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:06.983722 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/054d8392-289a-4eac-8926-ee85aaf15395-kube-api-access-blj95" pod "054d8392-289a-4eac-8926-ee85aaf15395" (UID: "054d8392-289a-4eac-8926-ee85aaf15395"). InnerVolumeSpecName "kube-api-access-blj95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:27:06.983889 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:06.983764 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/054d8392-289a-4eac-8926-ee85aaf15395-maas-api-xks-ca" pod "054d8392-289a-4eac-8926-ee85aaf15395" (UID: "054d8392-289a-4eac-8926-ee85aaf15395"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:27:07.083731 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:07.083694 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/054d8392-289a-4eac-8926-ee85aaf15395-maas-api-xks-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:27:07.083731 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:07.083726 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blj95\" (UniqueName: \"kubernetes.io/projected/054d8392-289a-4eac-8926-ee85aaf15395-kube-api-access-blj95\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:27:07.712138 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:07.712101 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9759f85a-lghlg" event={"ID":"054d8392-289a-4eac-8926-ee85aaf15395","Type":"ContainerDied","Data":"b9bd96da0c30db3a7c4da871720070a9b47465dd667962143a07458a8ce3ae4b"} Nov 03 14:27:07.712138 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:07.712129 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9759f85a-lghlg" Nov 03 14:27:07.712407 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:07.712149 5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9bd96da0c30db3a7c4da871720070a9b47465dd667962143a07458a8ce3ae4b" Nov 03 14:27:09.451447 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:27:09.451412 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:27:19.632605 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:19.632548 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc0cd4ad-vc5q8"] Nov 03 14:27:19.633365 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:19.633343 5060 cpu_manager.go:414] "containerMap: removing container" podUID="054d8392-289a-4eac-8926-ee85aaf15395" containerName="revoke-keys" Nov 03 14:27:19.633457 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:19.633367 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="054d8392-289a-4eac-8926-ee85aaf15395" containerName="revoke-keys" Nov 03 14:27:19.633554 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:19.633541 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="054d8392-289a-4eac-8926-ee85aaf15395" containerName="revoke-keys" Nov 03 14:27:19.652956 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:19.652895 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0cd4ad-vc5q8" Nov 03 14:27:19.687305 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:19.687266 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc0cd4ad-vc5q8"] Nov 03 14:27:19.738690 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:19.737992 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:27:19.738690 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:19.738558 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:27:19.738690 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:19.738650 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:27:19.739004 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:19.738931 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\"" Nov 03 14:27:19.770333 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:19.770283 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrc7v\" (UniqueName: \"kubernetes.io/projected/7b011271-14e8-41f9-b8e4-e978dbdacc27-kube-api-access-zrc7v\") pod \"maas-api-revoke-sub-bc0cd4ad-vc5q8\" (UID: \"7b011271-14e8-41f9-b8e4-e978dbdacc27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0cd4ad-vc5q8" Nov 03 14:27:19.770526 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:19.770404 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b011271-14e8-41f9-b8e4-e978dbdacc27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc0cd4ad-vc5q8\" (UID: \"7b011271-14e8-41f9-b8e4-e978dbdacc27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0cd4ad-vc5q8" Nov 03 14:27:19.770526 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:19.770498 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b011271-14e8-41f9-b8e4-e978dbdacc27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc0cd4ad-vc5q8\" (UID: \"7b011271-14e8-41f9-b8e4-e978dbdacc27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0cd4ad-vc5q8" Nov 03 14:27:19.872211 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:19.872170 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b011271-14e8-41f9-b8e4-e978dbdacc27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc0cd4ad-vc5q8\" (UID: \"7b011271-14e8-41f9-b8e4-e978dbdacc27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0cd4ad-vc5q8" Nov 03 14:27:19.872389 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:19.872352 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrc7v\" (UniqueName: \"kubernetes.io/projected/7b011271-14e8-41f9-b8e4-e978dbdacc27-kube-api-access-zrc7v\") pod \"maas-api-revoke-sub-bc0cd4ad-vc5q8\" (UID: \"7b011271-14e8-41f9-b8e4-e978dbdacc27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0cd4ad-vc5q8" Nov 03 14:27:19.872436 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:19.872423 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b011271-14e8-41f9-b8e4-e978dbdacc27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc0cd4ad-vc5q8\" (UID: \"7b011271-14e8-41f9-b8e4-e978dbdacc27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0cd4ad-vc5q8" Nov 03 14:27:19.873424 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:19.873393 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b011271-14e8-41f9-b8e4-e978dbdacc27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc0cd4ad-vc5q8\" (UID: \"7b011271-14e8-41f9-b8e4-e978dbdacc27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0cd4ad-vc5q8" Nov 03 14:27:19.875047 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:19.875021 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b011271-14e8-41f9-b8e4-e978dbdacc27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc0cd4ad-vc5q8\" (UID: \"7b011271-14e8-41f9-b8e4-e978dbdacc27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0cd4ad-vc5q8" Nov 03 14:27:19.904097 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:19.904021 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrc7v\" (UniqueName: \"kubernetes.io/projected/7b011271-14e8-41f9-b8e4-e978dbdacc27-kube-api-access-zrc7v\") pod \"maas-api-revoke-sub-bc0cd4ad-vc5q8\" (UID: \"7b011271-14e8-41f9-b8e4-e978dbdacc27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0cd4ad-vc5q8" Nov 03 14:27:20.056390 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:20.056353 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0cd4ad-vc5q8" Nov 03 14:27:20.385022 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:20.384990 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc0cd4ad-vc5q8"] Nov 03 14:27:20.386660 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:27:20.386631 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b011271_14e8_41f9_b8e4_e978dbdacc27.slice/crio-a1e16ef51ab87750cabf1ec64f6d3331fb57d31825680200dd43250cf71497eb WatchSource:0}: Error finding container a1e16ef51ab87750cabf1ec64f6d3331fb57d31825680200dd43250cf71497eb: Status 404 returned error can't find the container with id a1e16ef51ab87750cabf1ec64f6d3331fb57d31825680200dd43250cf71497eb Nov 03 14:27:20.908869 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:20.908813 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0cd4ad-vc5q8" event={"ID":"7b011271-14e8-41f9-b8e4-e978dbdacc27","Type":"ContainerStarted","Data":"74e4d3c483f51d6ece91b9736dc8f415f394130ad3861af4c91686cbc7ee1f4f"} Nov 03 14:27:20.909315 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:20.908881 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0cd4ad-vc5q8" event={"ID":"7b011271-14e8-41f9-b8e4-e978dbdacc27","Type":"ContainerStarted","Data":"a1e16ef51ab87750cabf1ec64f6d3331fb57d31825680200dd43250cf71497eb"} Nov 03 14:27:20.953859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:20.953792 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0cd4ad-vc5q8" podStartSLOduration=1.953779282 podStartE2EDuration="1.953779282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:27:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:27:20.950184482 +0000 UTC m=+974.838974063" watchObservedRunningTime="2026-11-03 14:27:20.953779282 +0000 UTC m=+974.842568862" Nov 03 14:27:21.917180 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:21.917143 5060 generic.go:356] "Generic (PLEG): container finished" podID="7b011271-14e8-41f9-b8e4-e978dbdacc27" containerID="74e4d3c483f51d6ece91b9736dc8f415f394130ad3861af4c91686cbc7ee1f4f" exitCode=0 Nov 03 14:27:21.917599 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:21.917226 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0cd4ad-vc5q8" event={"ID":"7b011271-14e8-41f9-b8e4-e978dbdacc27","Type":"ContainerDied","Data":"74e4d3c483f51d6ece91b9736dc8f415f394130ad3861af4c91686cbc7ee1f4f"} Nov 03 14:27:24.253873 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:24.244022 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0cd4ad-vc5q8" Nov 03 14:27:24.329141 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:24.329058 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b011271-14e8-41f9-b8e4-e978dbdacc27-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b011271-14e8-41f9-b8e4-e978dbdacc27-maas-api-openshift-ca\") pod \"7b011271-14e8-41f9-b8e4-e978dbdacc27\" (UID: \"7b011271-14e8-41f9-b8e4-e978dbdacc27\") " Nov 03 14:27:24.378097 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:24.377964 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b011271-14e8-41f9-b8e4-e978dbdacc27-maas-api-openshift-ca" pod "7b011271-14e8-41f9-b8e4-e978dbdacc27" (UID: "7b011271-14e8-41f9-b8e4-e978dbdacc27"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:27:24.378432 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:24.378406 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b011271-14e8-41f9-b8e4-e978dbdacc27-kube-api-access-zrc7v\" (UniqueName: \"kubernetes.io/projected/7b011271-14e8-41f9-b8e4-e978dbdacc27-kube-api-access-zrc7v\") pod \"7b011271-14e8-41f9-b8e4-e978dbdacc27\" (UID: \"7b011271-14e8-41f9-b8e4-e978dbdacc27\") " Nov 03 14:27:24.418856 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:24.418324 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b011271-14e8-41f9-b8e4-e978dbdacc27-kube-api-access-zrc7v" pod "7b011271-14e8-41f9-b8e4-e978dbdacc27" (UID: "7b011271-14e8-41f9-b8e4-e978dbdacc27"). InnerVolumeSpecName "kube-api-access-zrc7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:27:24.453694 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:24.445962 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b011271-14e8-41f9-b8e4-e978dbdacc27-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b011271-14e8-41f9-b8e4-e978dbdacc27-maas-api-xks-ca\") pod \"7b011271-14e8-41f9-b8e4-e978dbdacc27\" (UID: \"7b011271-14e8-41f9-b8e4-e978dbdacc27\") " Nov 03 14:27:24.453694 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:24.452124 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0cd4ad-vc5q8" event={"ID":"7b011271-14e8-41f9-b8e4-e978dbdacc27","Type":"ContainerDied","Data":"a1e16ef51ab87750cabf1ec64f6d3331fb57d31825680200dd43250cf71497eb"} Nov 03 14:27:24.453694 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:24.452185 5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1e16ef51ab87750cabf1ec64f6d3331fb57d31825680200dd43250cf71497eb" Nov 03 14:27:24.454921 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:24.454888 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b011271-14e8-41f9-b8e4-e978dbdacc27-maas-api-openshift-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:27:24.455157 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:24.455140 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrc7v\" (UniqueName: \"kubernetes.io/projected/7b011271-14e8-41f9-b8e4-e978dbdacc27-kube-api-access-zrc7v\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:27:24.477033 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:24.470441 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98" podUID="3aaaeea4-134f-4d17-a180-b4857c6425d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 14:27:24.523953 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:24.523908 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b011271-14e8-41f9-b8e4-e978dbdacc27-maas-api-xks-ca" pod "7b011271-14e8-41f9-b8e4-e978dbdacc27" (UID: "7b011271-14e8-41f9-b8e4-e978dbdacc27"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:27:24.563858 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:24.561168 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b011271-14e8-41f9-b8e4-e978dbdacc27-maas-api-xks-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:27:25.263656 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:25.263608 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0cd4ad-vc5q8" Nov 03 14:27:39.441653 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:27:39.441605 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:27:46.086919 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:46.086883 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6"] Nov 03 14:27:46.087378 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:46.087131 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" podUID="711e30e5-2c44-4295-974c-80a266623ae1" containerName="istio-proxy" containerID="cri-o://e74b2bfd81eb5d47beeb2ee25d5bd0af3f6d161ad1c4f4145f9f17dcd8e75039" gracePeriod=30 Nov 03 14:27:51.676225 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:51.676187 5060 generic.go:356] "Generic (PLEG): container finished" podID="711e30e5-2c44-4295-974c-80a266623ae1" containerID="e74b2bfd81eb5d47beeb2ee25d5bd0af3f6d161ad1c4f4145f9f17dcd8e75039" exitCode=0 Nov 03 14:27:51.802962 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:51.802917 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" event={"ID":"711e30e5-2c44-4295-974c-80a266623ae1","Type":"ContainerDied","Data":"e74b2bfd81eb5d47beeb2ee25d5bd0af3f6d161ad1c4f4145f9f17dcd8e75039"} Nov 03 14:27:51.828020 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:51.827982 5060 scope.go:122] "RemoveContainer" containerID="0ba1c7f2a6096b1f7b4f7d4f5e68ed118f027a5f9427f6aa40d4ed459eba6fab" Nov 03 14:27:52.371188 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:52.371158 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" Nov 03 14:27:52.423860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:52.417809 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/711e30e5-2c44-4295-974c-80a266623ae1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/711e30e5-2c44-4295-974c-80a266623ae1-workload-certs\") pod \"711e30e5-2c44-4295-974c-80a266623ae1\" (UID: \"711e30e5-2c44-4295-974c-80a266623ae1\") " Nov 03 14:27:52.423860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:52.417912 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/711e30e5-2c44-4295-974c-80a266623ae1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/711e30e5-2c44-4295-974c-80a266623ae1-istio-data\") pod \"711e30e5-2c44-4295-974c-80a266623ae1\" (UID: \"711e30e5-2c44-4295-974c-80a266623ae1\") " Nov 03 14:27:52.423860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:52.417957 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/711e30e5-2c44-4295-974c-80a266623ae1-istio-token\" (UniqueName: \"kubernetes.io/projected/711e30e5-2c44-4295-974c-80a266623ae1-istio-token\") pod \"711e30e5-2c44-4295-974c-80a266623ae1\" (UID: \"711e30e5-2c44-4295-974c-80a266623ae1\") " Nov 03 14:27:52.423860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:52.417991 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/711e30e5-2c44-4295-974c-80a266623ae1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/711e30e5-2c44-4295-974c-80a266623ae1-workload-socket\") pod \"711e30e5-2c44-4295-974c-80a266623ae1\" (UID: \"711e30e5-2c44-4295-974c-80a266623ae1\") " Nov 03 14:27:52.423860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:52.418025 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/711e30e5-2c44-4295-974c-80a266623ae1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/711e30e5-2c44-4295-974c-80a266623ae1-istio-envoy\") pod \"711e30e5-2c44-4295-974c-80a266623ae1\" (UID: \"711e30e5-2c44-4295-974c-80a266623ae1\") " Nov 03 14:27:52.423860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:52.418088 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/711e30e5-2c44-4295-974c-80a266623ae1-kube-api-access-fsb6p\" (UniqueName: \"kubernetes.io/projected/711e30e5-2c44-4295-974c-80a266623ae1-kube-api-access-fsb6p\") pod \"711e30e5-2c44-4295-974c-80a266623ae1\" (UID: \"711e30e5-2c44-4295-974c-80a266623ae1\") " Nov 03 14:27:52.423860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:52.418151 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/711e30e5-2c44-4295-974c-80a266623ae1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/711e30e5-2c44-4295-974c-80a266623ae1-credential-socket\") pod \"711e30e5-2c44-4295-974c-80a266623ae1\" (UID: \"711e30e5-2c44-4295-974c-80a266623ae1\") " Nov 03 14:27:52.423860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:52.418200 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/711e30e5-2c44-4295-974c-80a266623ae1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/711e30e5-2c44-4295-974c-80a266623ae1-istio-podinfo\") pod \"711e30e5-2c44-4295-974c-80a266623ae1\" (UID: \"711e30e5-2c44-4295-974c-80a266623ae1\") " Nov 03 14:27:52.423860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:52.418266 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/711e30e5-2c44-4295-974c-80a266623ae1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/711e30e5-2c44-4295-974c-80a266623ae1-istiod-ca-cert\") pod \"711e30e5-2c44-4295-974c-80a266623ae1\" (UID: \"711e30e5-2c44-4295-974c-80a266623ae1\") " Nov 03 14:27:52.424496 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:52.423926 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711e30e5-2c44-4295-974c-80a266623ae1-istio-data" pod "711e30e5-2c44-4295-974c-80a266623ae1" (UID: "711e30e5-2c44-4295-974c-80a266623ae1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:27:52.427854 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:52.425606 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/711e30e5-2c44-4295-974c-80a266623ae1-istiod-ca-cert" pod "711e30e5-2c44-4295-974c-80a266623ae1" (UID: "711e30e5-2c44-4295-974c-80a266623ae1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:27:52.427854 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:52.426917 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711e30e5-2c44-4295-974c-80a266623ae1-workload-socket" pod "711e30e5-2c44-4295-974c-80a266623ae1" (UID: "711e30e5-2c44-4295-974c-80a266623ae1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:27:52.427854 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:52.426981 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711e30e5-2c44-4295-974c-80a266623ae1-workload-certs" pod "711e30e5-2c44-4295-974c-80a266623ae1" (UID: "711e30e5-2c44-4295-974c-80a266623ae1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:27:52.427854 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:52.427027 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711e30e5-2c44-4295-974c-80a266623ae1-credential-socket" pod "711e30e5-2c44-4295-974c-80a266623ae1" (UID: "711e30e5-2c44-4295-974c-80a266623ae1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:27:52.442317 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:52.442245 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/711e30e5-2c44-4295-974c-80a266623ae1-istio-podinfo" pod "711e30e5-2c44-4295-974c-80a266623ae1" (UID: "711e30e5-2c44-4295-974c-80a266623ae1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:27:52.453815 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:52.453474 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/711e30e5-2c44-4295-974c-80a266623ae1-kube-api-access-fsb6p" pod "711e30e5-2c44-4295-974c-80a266623ae1" (UID: "711e30e5-2c44-4295-974c-80a266623ae1"). InnerVolumeSpecName "kube-api-access-fsb6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:27:52.454190 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:52.454147 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711e30e5-2c44-4295-974c-80a266623ae1-istio-envoy" pod "711e30e5-2c44-4295-974c-80a266623ae1" (UID: "711e30e5-2c44-4295-974c-80a266623ae1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:27:52.458404 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:52.458351 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/711e30e5-2c44-4295-974c-80a266623ae1-istio-token" pod "711e30e5-2c44-4295-974c-80a266623ae1" (UID: "711e30e5-2c44-4295-974c-80a266623ae1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:27:52.530930 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:52.525666 5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/711e30e5-2c44-4295-974c-80a266623ae1-istiod-ca-cert\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:27:52.530930 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:52.525712 5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/711e30e5-2c44-4295-974c-80a266623ae1-workload-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:27:52.530930 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:52.525727 5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/711e30e5-2c44-4295-974c-80a266623ae1-istio-data\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:27:52.530930 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:52.525740 5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/711e30e5-2c44-4295-974c-80a266623ae1-istio-token\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:27:52.530930 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:52.525753 5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/711e30e5-2c44-4295-974c-80a266623ae1-workload-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:27:52.530930 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:52.525767 5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/711e30e5-2c44-4295-974c-80a266623ae1-istio-envoy\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:27:52.530930 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:52.525784 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsb6p\" (UniqueName: \"kubernetes.io/projected/711e30e5-2c44-4295-974c-80a266623ae1-kube-api-access-fsb6p\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:27:52.530930 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:52.525800 5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/711e30e5-2c44-4295-974c-80a266623ae1-credential-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:27:52.530930 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:52.525813 5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/711e30e5-2c44-4295-974c-80a266623ae1-istio-podinfo\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:27:53.023796 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:53.023752 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" event={"ID":"711e30e5-2c44-4295-974c-80a266623ae1","Type":"ContainerDied","Data":"996b2b17a76d765c4dbcd2207117b1cab8978f625a9388ce991d0e725ddd0c66"} Nov 03 14:27:53.030950 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:53.030922 5060 scope.go:122] "RemoveContainer" containerID="e74b2bfd81eb5d47beeb2ee25d5bd0af3f6d161ad1c4f4145f9f17dcd8e75039" Nov 03 14:27:53.031266 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:53.031245 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6" Nov 03 14:27:53.432856 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:53.424237 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6"] Nov 03 14:27:53.462603 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:53.445429 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-46nz6"] Nov 03 14:27:54.253411 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:54.253370 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="711e30e5-2c44-4295-974c-80a266623ae1" path="/var/lib/kubelet/pods/711e30e5-2c44-4295-974c-80a266623ae1/volumes" Nov 03 14:27:59.046132 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:59.046097 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91aabadb-gwns8"] Nov 03 14:27:59.058624 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:27:59.058593 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91aabadb-gwns8"] Nov 03 14:28:00.250424 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:00.250391 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f29f3d1-2c91-4403-ac7c-f404fb688ac4" path="/var/lib/kubelet/pods/8f29f3d1-2c91-4403-ac7c-f404fb688ac4/volumes" Nov 03 14:28:09.447362 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:28:09.447321 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:28:13.587152 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:13.587119 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw"] Nov 03 14:28:13.587612 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:13.587367 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" podUID="c60d54c8-1f41-4f9c-9401-63a07e12f664" containerName="istio-proxy" containerID="cri-o://fab7c2a973e48ae33680ceaa5e3a19186522999f512ae1903ced1b5d1cb65432" gracePeriod=30 Nov 03 14:28:16.069439 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:16.069397 5060 patch_prober.go:29] interesting pod/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 14:28:16.069829 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:16.069479 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" podUID="c60d54c8-1f41-4f9c-9401-63a07e12f664" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 14:28:18.880573 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:18.880542 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" Nov 03 14:28:18.907226 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:18.907196 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c60d54c8-1f41-4f9c-9401-63a07e12f664-istio-token\" (UniqueName: \"kubernetes.io/projected/c60d54c8-1f41-4f9c-9401-63a07e12f664-istio-token\") pod \"c60d54c8-1f41-4f9c-9401-63a07e12f664\" (UID: \"c60d54c8-1f41-4f9c-9401-63a07e12f664\") " Nov 03 14:28:18.907226 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:18.907229 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c60d54c8-1f41-4f9c-9401-63a07e12f664-kube-api-access-wsppv\" (UniqueName: \"kubernetes.io/projected/c60d54c8-1f41-4f9c-9401-63a07e12f664-kube-api-access-wsppv\") pod \"c60d54c8-1f41-4f9c-9401-63a07e12f664\" (UID: \"c60d54c8-1f41-4f9c-9401-63a07e12f664\") " Nov 03 14:28:18.907474 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:18.907260 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c60d54c8-1f41-4f9c-9401-63a07e12f664-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c60d54c8-1f41-4f9c-9401-63a07e12f664-istio-data\") pod \"c60d54c8-1f41-4f9c-9401-63a07e12f664\" (UID: \"c60d54c8-1f41-4f9c-9401-63a07e12f664\") " Nov 03 14:28:18.907474 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:18.907389 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c60d54c8-1f41-4f9c-9401-63a07e12f664-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c60d54c8-1f41-4f9c-9401-63a07e12f664-istio-envoy\") pod \"c60d54c8-1f41-4f9c-9401-63a07e12f664\" (UID: \"c60d54c8-1f41-4f9c-9401-63a07e12f664\") " Nov 03 14:28:18.907474 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:18.907436 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c60d54c8-1f41-4f9c-9401-63a07e12f664-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c60d54c8-1f41-4f9c-9401-63a07e12f664-workload-socket\") pod \"c60d54c8-1f41-4f9c-9401-63a07e12f664\" (UID: \"c60d54c8-1f41-4f9c-9401-63a07e12f664\") " Nov 03 14:28:18.907624 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:18.907500 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c60d54c8-1f41-4f9c-9401-63a07e12f664-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c60d54c8-1f41-4f9c-9401-63a07e12f664-workload-certs\") pod \"c60d54c8-1f41-4f9c-9401-63a07e12f664\" (UID: \"c60d54c8-1f41-4f9c-9401-63a07e12f664\") " Nov 03 14:28:18.907624 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:18.907554 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c60d54c8-1f41-4f9c-9401-63a07e12f664-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c60d54c8-1f41-4f9c-9401-63a07e12f664-credential-socket\") pod \"c60d54c8-1f41-4f9c-9401-63a07e12f664\" (UID: \"c60d54c8-1f41-4f9c-9401-63a07e12f664\") " Nov 03 14:28:18.907624 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:18.907566 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60d54c8-1f41-4f9c-9401-63a07e12f664-istio-data" pod "c60d54c8-1f41-4f9c-9401-63a07e12f664" (UID: "c60d54c8-1f41-4f9c-9401-63a07e12f664"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:28:18.907624 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:18.907606 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c60d54c8-1f41-4f9c-9401-63a07e12f664-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c60d54c8-1f41-4f9c-9401-63a07e12f664-istiod-ca-cert\") pod \"c60d54c8-1f41-4f9c-9401-63a07e12f664\" (UID: \"c60d54c8-1f41-4f9c-9401-63a07e12f664\") " Nov 03 14:28:18.907818 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:18.907782 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60d54c8-1f41-4f9c-9401-63a07e12f664-workload-certs" pod "c60d54c8-1f41-4f9c-9401-63a07e12f664" (UID: "c60d54c8-1f41-4f9c-9401-63a07e12f664"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:28:18.907818 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:18.907785 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60d54c8-1f41-4f9c-9401-63a07e12f664-workload-socket" pod "c60d54c8-1f41-4f9c-9401-63a07e12f664" (UID: "c60d54c8-1f41-4f9c-9401-63a07e12f664"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:28:18.907818 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:18.907786 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c60d54c8-1f41-4f9c-9401-63a07e12f664-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c60d54c8-1f41-4f9c-9401-63a07e12f664-istio-podinfo\") pod \"c60d54c8-1f41-4f9c-9401-63a07e12f664\" (UID: \"c60d54c8-1f41-4f9c-9401-63a07e12f664\") " Nov 03 14:28:18.908009 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:18.907978 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60d54c8-1f41-4f9c-9401-63a07e12f664-credential-socket" pod "c60d54c8-1f41-4f9c-9401-63a07e12f664" (UID: "c60d54c8-1f41-4f9c-9401-63a07e12f664"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:28:18.908176 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:18.908145 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c60d54c8-1f41-4f9c-9401-63a07e12f664-istiod-ca-cert" pod "c60d54c8-1f41-4f9c-9401-63a07e12f664" (UID: "c60d54c8-1f41-4f9c-9401-63a07e12f664"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:28:18.908368 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:18.908349 5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c60d54c8-1f41-4f9c-9401-63a07e12f664-istio-data\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:28:18.908473 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:18.908376 5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c60d54c8-1f41-4f9c-9401-63a07e12f664-workload-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:28:18.908473 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:18.908391 5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c60d54c8-1f41-4f9c-9401-63a07e12f664-workload-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:28:18.908473 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:18.908405 5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c60d54c8-1f41-4f9c-9401-63a07e12f664-credential-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:28:18.908473 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:18.908418 5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c60d54c8-1f41-4f9c-9401-63a07e12f664-istiod-ca-cert\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:28:18.910475 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:18.910448 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c60d54c8-1f41-4f9c-9401-63a07e12f664-istio-podinfo" pod "c60d54c8-1f41-4f9c-9401-63a07e12f664" (UID: "c60d54c8-1f41-4f9c-9401-63a07e12f664"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:28:18.910475 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:18.910454 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60d54c8-1f41-4f9c-9401-63a07e12f664-istio-envoy" pod "c60d54c8-1f41-4f9c-9401-63a07e12f664" (UID: "c60d54c8-1f41-4f9c-9401-63a07e12f664"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:28:18.910607 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:18.910483 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c60d54c8-1f41-4f9c-9401-63a07e12f664-kube-api-access-wsppv" pod "c60d54c8-1f41-4f9c-9401-63a07e12f664" (UID: "c60d54c8-1f41-4f9c-9401-63a07e12f664"). InnerVolumeSpecName "kube-api-access-wsppv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:28:18.910607 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:18.910513 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c60d54c8-1f41-4f9c-9401-63a07e12f664-istio-token" pod "c60d54c8-1f41-4f9c-9401-63a07e12f664" (UID: "c60d54c8-1f41-4f9c-9401-63a07e12f664"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:28:19.009690 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:19.009657 5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c60d54c8-1f41-4f9c-9401-63a07e12f664-istio-podinfo\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:28:19.009690 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:19.009689 5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c60d54c8-1f41-4f9c-9401-63a07e12f664-istio-token\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:28:19.009921 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:19.009701 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsppv\" (UniqueName: \"kubernetes.io/projected/c60d54c8-1f41-4f9c-9401-63a07e12f664-kube-api-access-wsppv\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:28:19.009921 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:19.009716 5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c60d54c8-1f41-4f9c-9401-63a07e12f664-istio-envoy\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:28:19.545929 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:19.537178 5060 generic.go:356] "Generic (PLEG): container finished" podID="c60d54c8-1f41-4f9c-9401-63a07e12f664" containerID="fab7c2a973e48ae33680ceaa5e3a19186522999f512ae1903ced1b5d1cb65432" exitCode=0 Nov 03 14:28:19.545929 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:19.537363 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" event={"ID":"c60d54c8-1f41-4f9c-9401-63a07e12f664","Type":"ContainerDied","Data":"fab7c2a973e48ae33680ceaa5e3a19186522999f512ae1903ced1b5d1cb65432"} Nov 03 14:28:19.545929 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:19.537420 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" event={"ID":"c60d54c8-1f41-4f9c-9401-63a07e12f664","Type":"ContainerDied","Data":"f209e8f2594b60380867e8015eb1d40453d4a78ff19321e13ac60d9224ad3e36"} Nov 03 14:28:19.545929 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:19.537455 5060 scope.go:122] "RemoveContainer" containerID="fab7c2a973e48ae33680ceaa5e3a19186522999f512ae1903ced1b5d1cb65432" Nov 03 14:28:19.545929 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:19.537631 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw" Nov 03 14:28:19.594995 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:19.594959 5060 scope.go:122] "RemoveContainer" containerID="1bd3a1b6dec9eb61ece036757e428efd0cfc141931c5ae8d2595dc0166c1c964" Nov 03 14:28:19.610222 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:19.610135 5060 scope.go:122] "RemoveContainer" containerID="fab7c2a973e48ae33680ceaa5e3a19186522999f512ae1903ced1b5d1cb65432" Nov 03 14:28:19.617734 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:28:19.617396 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fab7c2a973e48ae33680ceaa5e3a19186522999f512ae1903ced1b5d1cb65432\": container with ID starting with fab7c2a973e48ae33680ceaa5e3a19186522999f512ae1903ced1b5d1cb65432 not found: ID does not exist" containerID="fab7c2a973e48ae33680ceaa5e3a19186522999f512ae1903ced1b5d1cb65432" Nov 03 14:28:19.617734 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:19.617436 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fab7c2a973e48ae33680ceaa5e3a19186522999f512ae1903ced1b5d1cb65432"} err="failed to get container status \"fab7c2a973e48ae33680ceaa5e3a19186522999f512ae1903ced1b5d1cb65432\": rpc error: code = NotFound desc = could not find container \"fab7c2a973e48ae33680ceaa5e3a19186522999f512ae1903ced1b5d1cb65432\": container with ID starting with fab7c2a973e48ae33680ceaa5e3a19186522999f512ae1903ced1b5d1cb65432 not found: ID does not exist" Nov 03 14:28:19.617734 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:19.617463 5060 scope.go:122] "RemoveContainer" containerID="1bd3a1b6dec9eb61ece036757e428efd0cfc141931c5ae8d2595dc0166c1c964" Nov 03 14:28:19.618236 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:28:19.618143 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bd3a1b6dec9eb61ece036757e428efd0cfc141931c5ae8d2595dc0166c1c964\": container with ID starting with 1bd3a1b6dec9eb61ece036757e428efd0cfc141931c5ae8d2595dc0166c1c964 not found: ID does not exist" containerID="1bd3a1b6dec9eb61ece036757e428efd0cfc141931c5ae8d2595dc0166c1c964" Nov 03 14:28:19.618236 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:19.618177 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bd3a1b6dec9eb61ece036757e428efd0cfc141931c5ae8d2595dc0166c1c964"} err="failed to get container status \"1bd3a1b6dec9eb61ece036757e428efd0cfc141931c5ae8d2595dc0166c1c964\": rpc error: code = NotFound desc = could not find container \"1bd3a1b6dec9eb61ece036757e428efd0cfc141931c5ae8d2595dc0166c1c964\": container with ID starting with 1bd3a1b6dec9eb61ece036757e428efd0cfc141931c5ae8d2595dc0166c1c964 not found: ID does not exist" Nov 03 14:28:19.633788 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:19.633738 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw"] Nov 03 14:28:19.639856 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:19.639810 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-l7nnw"] Nov 03 14:28:20.290183 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:20.290147 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c60d54c8-1f41-4f9c-9401-63a07e12f664" path="/var/lib/kubelet/pods/c60d54c8-1f41-4f9c-9401-63a07e12f664/volumes" Nov 03 14:28:22.188861 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:22.187897 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-655b301b-brlf2"] Nov 03 14:28:22.188861 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:22.188804 5060 cpu_manager.go:414] "containerMap: removing container" podUID="711e30e5-2c44-4295-974c-80a266623ae1" containerName="istio-proxy" Nov 03 14:28:22.188861 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:22.188825 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="711e30e5-2c44-4295-974c-80a266623ae1" containerName="istio-proxy" Nov 03 14:28:22.188861 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:22.188855 5060 cpu_manager.go:414] "containerMap: removing container" podUID="7b011271-14e8-41f9-b8e4-e978dbdacc27" containerName="revoke-keys" Nov 03 14:28:22.188861 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:22.188863 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b011271-14e8-41f9-b8e4-e978dbdacc27" containerName="revoke-keys" Nov 03 14:28:22.189668 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:22.188882 5060 cpu_manager.go:414] "containerMap: removing container" podUID="711e30e5-2c44-4295-974c-80a266623ae1" containerName="istio-proxy" Nov 03 14:28:22.189668 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:22.188890 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="711e30e5-2c44-4295-974c-80a266623ae1" containerName="istio-proxy" Nov 03 14:28:22.189668 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:22.188905 5060 cpu_manager.go:414] "containerMap: removing container" podUID="c60d54c8-1f41-4f9c-9401-63a07e12f664" containerName="istio-proxy" Nov 03 14:28:22.189668 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:22.188912 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c60d54c8-1f41-4f9c-9401-63a07e12f664" containerName="istio-proxy" Nov 03 14:28:22.189668 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:22.189157 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="711e30e5-2c44-4295-974c-80a266623ae1" containerName="istio-proxy" Nov 03 14:28:22.189668 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:22.189172 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b011271-14e8-41f9-b8e4-e978dbdacc27" containerName="revoke-keys" Nov 03 14:28:22.189668 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:22.189183 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="711e30e5-2c44-4295-974c-80a266623ae1" containerName="istio-proxy" Nov 03 14:28:22.189668 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:22.189193 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c60d54c8-1f41-4f9c-9401-63a07e12f664" containerName="istio-proxy" Nov 03 14:28:22.189668 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:22.189202 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c60d54c8-1f41-4f9c-9401-63a07e12f664" containerName="istio-proxy" Nov 03 14:28:22.229857 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:22.209541 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655b301b-brlf2" Nov 03 14:28:22.229857 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:22.227027 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-655b301b-brlf2"] Nov 03 14:28:22.253860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:22.251489 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:28:22.278048 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:22.267109 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:28:22.278048 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:22.267229 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\"" Nov 03 14:28:22.278048 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:22.267323 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:28:22.419875 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:22.363193 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-655b301b-brlf2\" (UID: \"af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655b301b-brlf2" Nov 03 14:28:22.419875 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:22.363318 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-655b301b-brlf2\" (UID: \"af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655b301b-brlf2" Nov 03 14:28:22.419875 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:22.363362 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92b6v\" (UniqueName: \"kubernetes.io/projected/af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1-kube-api-access-92b6v\") pod \"maas-api-revoke-sub-655b301b-brlf2\" (UID: \"af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655b301b-brlf2" Nov 03 14:28:22.432538 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:22.432490 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98" podUID="3aaaeea4-134f-4d17-a180-b4857c6425d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 14:28:22.568795 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:22.568756 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-655b301b-brlf2\" (UID: \"af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655b301b-brlf2" Nov 03 14:28:22.594156 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:22.594122 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-655b301b-brlf2\" (UID: \"af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655b301b-brlf2" Nov 03 14:28:22.648856 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:22.648085 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92b6v\" (UniqueName: \"kubernetes.io/projected/af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1-kube-api-access-92b6v\") pod \"maas-api-revoke-sub-655b301b-brlf2\" (UID: \"af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655b301b-brlf2" Nov 03 14:28:22.721186 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:22.721148 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-655b301b-brlf2\" (UID: \"af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655b301b-brlf2" Nov 03 14:28:22.723025 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:22.722996 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-655b301b-brlf2\" (UID: \"af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655b301b-brlf2" Nov 03 14:28:22.841922 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:22.827600 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92b6v\" (UniqueName: \"kubernetes.io/projected/af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1-kube-api-access-92b6v\") pod \"maas-api-revoke-sub-655b301b-brlf2\" (UID: \"af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655b301b-brlf2" Nov 03 14:28:22.900477 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:22.900164 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655b301b-brlf2" Nov 03 14:28:23.319045 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:23.319009 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-655b301b-brlf2"] Nov 03 14:28:23.325151 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:28:23.325115 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf5d2cef_69d9_4ed2_a46c_b4d84f40d7e1.slice/crio-4729e28e4b69f468106ec9e2e16605f37763aca0766b9fb54b79f64f93658b07 WatchSource:0}: Error finding container 4729e28e4b69f468106ec9e2e16605f37763aca0766b9fb54b79f64f93658b07: Status 404 returned error can't find the container with id 4729e28e4b69f468106ec9e2e16605f37763aca0766b9fb54b79f64f93658b07 Nov 03 14:28:23.940694 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:23.940660 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655b301b-brlf2" event={"ID":"af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1","Type":"ContainerStarted","Data":"9fef1ba9e2e5c153b9e65ee9fa84e1fa7b47a450fe0f7f44eafe242c41a4c6a0"} Nov 03 14:28:23.940907 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:23.940723 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655b301b-brlf2" event={"ID":"af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1","Type":"ContainerStarted","Data":"4729e28e4b69f468106ec9e2e16605f37763aca0766b9fb54b79f64f93658b07"} Nov 03 14:28:23.984979 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:23.984807 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655b301b-brlf2" podStartSLOduration=1.9847939430000001 podStartE2EDuration="1.984793943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:28:23.984599982 +0000 UTC m=+1037.873389564" watchObservedRunningTime="2026-11-03 14:28:23.984793943 +0000 UTC m=+1037.873583522" Nov 03 14:28:24.949384 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:24.949345 5060 generic.go:356] "Generic (PLEG): container finished" podID="af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1" containerID="9fef1ba9e2e5c153b9e65ee9fa84e1fa7b47a450fe0f7f44eafe242c41a4c6a0" exitCode=0 Nov 03 14:28:24.949775 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:24.949433 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655b301b-brlf2" event={"ID":"af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1","Type":"ContainerDied","Data":"9fef1ba9e2e5c153b9e65ee9fa84e1fa7b47a450fe0f7f44eafe242c41a4c6a0"} Nov 03 14:28:26.103374 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:26.103348 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655b301b-brlf2" Nov 03 14:28:26.223154 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:26.223055 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1-maas-api-xks-ca\") pod \"af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1\" (UID: \"af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1\") " Nov 03 14:28:26.223337 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:26.223159 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1-maas-api-openshift-ca\") pod \"af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1\" (UID: \"af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1\") " Nov 03 14:28:26.223396 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:26.223349 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1-kube-api-access-92b6v\" (UniqueName: \"kubernetes.io/projected/af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1-kube-api-access-92b6v\") pod \"af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1\" (UID: \"af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1\") " Nov 03 14:28:26.223574 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:26.223551 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1-maas-api-openshift-ca" pod "af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1" (UID: "af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:28:26.223750 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:26.223734 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1-maas-api-openshift-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:28:26.225140 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:26.225116 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1-maas-api-xks-ca" pod "af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1" (UID: "af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:28:26.225392 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:26.225372 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1-kube-api-access-92b6v" pod "af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1" (UID: "af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1"). InnerVolumeSpecName "kube-api-access-92b6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:28:26.325267 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:26.325228 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92b6v\" (UniqueName: \"kubernetes.io/projected/af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1-kube-api-access-92b6v\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:28:26.325267 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:26.325267 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1-maas-api-xks-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:28:26.974315 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:26.974285 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655b301b-brlf2" Nov 03 14:28:26.974492 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:26.974279 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655b301b-brlf2" event={"ID":"af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1","Type":"ContainerDied","Data":"4729e28e4b69f468106ec9e2e16605f37763aca0766b9fb54b79f64f93658b07"} Nov 03 14:28:26.974492 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:26.974454 5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4729e28e4b69f468106ec9e2e16605f37763aca0766b9fb54b79f64f93658b07" Nov 03 14:28:29.040638 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:29.040599 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-041fbab7-4pp7d"] Nov 03 14:28:29.052920 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:29.052890 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-041fbab7-4pp7d"] Nov 03 14:28:30.249348 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:30.249318 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="683f2838-8f53-4f9e-8bb5-a156ca3ff2d8" path="/var/lib/kubelet/pods/683f2838-8f53-4f9e-8bb5-a156ca3ff2d8/volumes" Nov 03 14:28:36.250855 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:36.250804 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0db7e167-6hrgl"] Nov 03 14:28:36.251262 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:36.251247 5060 cpu_manager.go:414] "containerMap: removing container" podUID="af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1" containerName="revoke-keys" Nov 03 14:28:36.251314 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:36.251264 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1" containerName="revoke-keys" Nov 03 14:28:36.251314 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:36.251279 5060 cpu_manager.go:414] "containerMap: removing container" podUID="c60d54c8-1f41-4f9c-9401-63a07e12f664" containerName="istio-proxy" Nov 03 14:28:36.251314 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:36.251284 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c60d54c8-1f41-4f9c-9401-63a07e12f664" containerName="istio-proxy" Nov 03 14:28:36.251411 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:36.251383 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1" containerName="revoke-keys" Nov 03 14:28:36.255856 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:36.255822 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db7e167-6hrgl" Nov 03 14:28:36.272723 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:36.272698 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0db7e167-6hrgl"] Nov 03 14:28:36.282058 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:36.282035 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:28:36.282183 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:36.282079 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:28:36.282183 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:36.282116 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\"" Nov 03 14:28:36.284537 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:36.284517 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:28:36.324559 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:36.324523 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f5389c9-92e4-41ec-bd3f-c4fc4723c51c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0db7e167-6hrgl\" (UID: \"5f5389c9-92e4-41ec-bd3f-c4fc4723c51c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db7e167-6hrgl" Nov 03 14:28:36.324738 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:36.324568 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f5389c9-92e4-41ec-bd3f-c4fc4723c51c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0db7e167-6hrgl\" (UID: \"5f5389c9-92e4-41ec-bd3f-c4fc4723c51c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db7e167-6hrgl" Nov 03 14:28:36.324738 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:36.324705 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxb8j\" (UniqueName: \"kubernetes.io/projected/5f5389c9-92e4-41ec-bd3f-c4fc4723c51c-kube-api-access-dxb8j\") pod \"maas-api-revoke-sub-0db7e167-6hrgl\" (UID: \"5f5389c9-92e4-41ec-bd3f-c4fc4723c51c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db7e167-6hrgl" Nov 03 14:28:36.425590 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:36.425535 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f5389c9-92e4-41ec-bd3f-c4fc4723c51c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0db7e167-6hrgl\" (UID: \"5f5389c9-92e4-41ec-bd3f-c4fc4723c51c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db7e167-6hrgl" Nov 03 14:28:36.425797 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:36.425604 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f5389c9-92e4-41ec-bd3f-c4fc4723c51c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0db7e167-6hrgl\" (UID: \"5f5389c9-92e4-41ec-bd3f-c4fc4723c51c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db7e167-6hrgl" Nov 03 14:28:36.425797 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:36.425667 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxb8j\" (UniqueName: \"kubernetes.io/projected/5f5389c9-92e4-41ec-bd3f-c4fc4723c51c-kube-api-access-dxb8j\") pod \"maas-api-revoke-sub-0db7e167-6hrgl\" (UID: \"5f5389c9-92e4-41ec-bd3f-c4fc4723c51c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db7e167-6hrgl" Nov 03 14:28:36.426783 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:36.426755 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f5389c9-92e4-41ec-bd3f-c4fc4723c51c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0db7e167-6hrgl\" (UID: \"5f5389c9-92e4-41ec-bd3f-c4fc4723c51c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db7e167-6hrgl" Nov 03 14:28:36.429213 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:36.429175 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f5389c9-92e4-41ec-bd3f-c4fc4723c51c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0db7e167-6hrgl\" (UID: \"5f5389c9-92e4-41ec-bd3f-c4fc4723c51c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db7e167-6hrgl" Nov 03 14:28:36.448292 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:36.448248 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxb8j\" (UniqueName: \"kubernetes.io/projected/5f5389c9-92e4-41ec-bd3f-c4fc4723c51c-kube-api-access-dxb8j\") pod \"maas-api-revoke-sub-0db7e167-6hrgl\" (UID: \"5f5389c9-92e4-41ec-bd3f-c4fc4723c51c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db7e167-6hrgl" Nov 03 14:28:36.568928 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:36.568821 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db7e167-6hrgl" Nov 03 14:28:36.902970 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:36.902876 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0db7e167-6hrgl"] Nov 03 14:28:37.104470 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:37.104430 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db7e167-6hrgl" event={"ID":"5f5389c9-92e4-41ec-bd3f-c4fc4723c51c","Type":"ContainerStarted","Data":"5a52f2e3a3b6cc039820d0b94eb3d6f03be720974f4f5ad5e79aff51b8666b89"} Nov 03 14:28:38.112698 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:38.112664 5060 generic.go:356] "Generic (PLEG): container finished" podID="5f5389c9-92e4-41ec-bd3f-c4fc4723c51c" containerID="77dd79ede176cfd2856a7a353b7f9972ce1cf89de321cdf80f0b5992bab0b678" exitCode=0 Nov 03 14:28:38.113113 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:38.112750 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db7e167-6hrgl" event={"ID":"5f5389c9-92e4-41ec-bd3f-c4fc4723c51c","Type":"ContainerDied","Data":"77dd79ede176cfd2856a7a353b7f9972ce1cf89de321cdf80f0b5992bab0b678"} Nov 03 14:28:39.280305 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:39.280269 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db7e167-6hrgl" Nov 03 14:28:39.396215 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:39.396065 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f5389c9-92e4-41ec-bd3f-c4fc4723c51c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f5389c9-92e4-41ec-bd3f-c4fc4723c51c-maas-api-xks-ca\") pod \"5f5389c9-92e4-41ec-bd3f-c4fc4723c51c\" (UID: \"5f5389c9-92e4-41ec-bd3f-c4fc4723c51c\") " Nov 03 14:28:39.396921 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:39.396886 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f5389c9-92e4-41ec-bd3f-c4fc4723c51c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f5389c9-92e4-41ec-bd3f-c4fc4723c51c-maas-api-openshift-ca\") pod \"5f5389c9-92e4-41ec-bd3f-c4fc4723c51c\" (UID: \"5f5389c9-92e4-41ec-bd3f-c4fc4723c51c\") " Nov 03 14:28:39.397228 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:39.397197 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f5389c9-92e4-41ec-bd3f-c4fc4723c51c-kube-api-access-dxb8j\" (UniqueName: \"kubernetes.io/projected/5f5389c9-92e4-41ec-bd3f-c4fc4723c51c-kube-api-access-dxb8j\") pod \"5f5389c9-92e4-41ec-bd3f-c4fc4723c51c\" (UID: \"5f5389c9-92e4-41ec-bd3f-c4fc4723c51c\") " Nov 03 14:28:39.398669 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:39.398646 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f5389c9-92e4-41ec-bd3f-c4fc4723c51c-maas-api-openshift-ca" pod "5f5389c9-92e4-41ec-bd3f-c4fc4723c51c" (UID: "5f5389c9-92e4-41ec-bd3f-c4fc4723c51c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:28:39.401315 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:39.401279 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f5389c9-92e4-41ec-bd3f-c4fc4723c51c-kube-api-access-dxb8j" pod "5f5389c9-92e4-41ec-bd3f-c4fc4723c51c" (UID: "5f5389c9-92e4-41ec-bd3f-c4fc4723c51c"). InnerVolumeSpecName "kube-api-access-dxb8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:28:39.401815 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:39.401791 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f5389c9-92e4-41ec-bd3f-c4fc4723c51c-maas-api-xks-ca" pod "5f5389c9-92e4-41ec-bd3f-c4fc4723c51c" (UID: "5f5389c9-92e4-41ec-bd3f-c4fc4723c51c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:28:39.459110 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:28:39.459069 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:28:39.499406 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:39.499370 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f5389c9-92e4-41ec-bd3f-c4fc4723c51c-maas-api-xks-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:28:39.499406 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:39.499409 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f5389c9-92e4-41ec-bd3f-c4fc4723c51c-maas-api-openshift-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:28:39.499628 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:39.499425 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxb8j\" (UniqueName: \"kubernetes.io/projected/5f5389c9-92e4-41ec-bd3f-c4fc4723c51c-kube-api-access-dxb8j\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:28:40.127455 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:40.127424 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db7e167-6hrgl" event={"ID":"5f5389c9-92e4-41ec-bd3f-c4fc4723c51c","Type":"ContainerDied","Data":"5a52f2e3a3b6cc039820d0b94eb3d6f03be720974f4f5ad5e79aff51b8666b89"} Nov 03 14:28:40.127629 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:40.127472 5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a52f2e3a3b6cc039820d0b94eb3d6f03be720974f4f5ad5e79aff51b8666b89" Nov 03 14:28:40.127629 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:40.127434 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db7e167-6hrgl" Nov 03 14:28:40.567518 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:40.567479 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0e0e478-hmwl5"] Nov 03 14:28:40.568213 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:40.568191 5060 cpu_manager.go:414] "containerMap: removing container" podUID="5f5389c9-92e4-41ec-bd3f-c4fc4723c51c" containerName="revoke-keys" Nov 03 14:28:40.568313 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:40.568214 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f5389c9-92e4-41ec-bd3f-c4fc4723c51c" containerName="revoke-keys" Nov 03 14:28:40.568394 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:40.568379 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f5389c9-92e4-41ec-bd3f-c4fc4723c51c" containerName="revoke-keys" Nov 03 14:28:40.571722 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:40.571700 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e0e478-hmwl5" Nov 03 14:28:40.591450 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:40.591417 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:28:40.591631 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:40.591611 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\"" Nov 03 14:28:40.594918 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:40.594895 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0e0e478-hmwl5"] Nov 03 14:28:40.603464 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:40.603438 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:28:40.603605 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:40.603487 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:28:40.714356 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:40.714321 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dc68081-80ec-4fc5-81c8-ec0b96a4647a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0e0e478-hmwl5\" (UID: \"9dc68081-80ec-4fc5-81c8-ec0b96a4647a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e0e478-hmwl5" Nov 03 14:28:40.714533 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:40.714364 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dc68081-80ec-4fc5-81c8-ec0b96a4647a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0e0e478-hmwl5\" (UID: \"9dc68081-80ec-4fc5-81c8-ec0b96a4647a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e0e478-hmwl5" Nov 03 14:28:40.714533 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:40.714395 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv876\" (UniqueName: \"kubernetes.io/projected/9dc68081-80ec-4fc5-81c8-ec0b96a4647a-kube-api-access-zv876\") pod \"maas-api-revoke-sub-b0e0e478-hmwl5\" (UID: \"9dc68081-80ec-4fc5-81c8-ec0b96a4647a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e0e478-hmwl5" Nov 03 14:28:40.815639 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:40.815597 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dc68081-80ec-4fc5-81c8-ec0b96a4647a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0e0e478-hmwl5\" (UID: \"9dc68081-80ec-4fc5-81c8-ec0b96a4647a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e0e478-hmwl5" Nov 03 14:28:40.815857 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:40.815645 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dc68081-80ec-4fc5-81c8-ec0b96a4647a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0e0e478-hmwl5\" (UID: \"9dc68081-80ec-4fc5-81c8-ec0b96a4647a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e0e478-hmwl5" Nov 03 14:28:40.815857 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:40.815708 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv876\" (UniqueName: \"kubernetes.io/projected/9dc68081-80ec-4fc5-81c8-ec0b96a4647a-kube-api-access-zv876\") pod \"maas-api-revoke-sub-b0e0e478-hmwl5\" (UID: \"9dc68081-80ec-4fc5-81c8-ec0b96a4647a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e0e478-hmwl5" Nov 03 14:28:40.816389 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:40.816363 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dc68081-80ec-4fc5-81c8-ec0b96a4647a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0e0e478-hmwl5\" (UID: \"9dc68081-80ec-4fc5-81c8-ec0b96a4647a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e0e478-hmwl5" Nov 03 14:28:40.818284 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:40.818219 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dc68081-80ec-4fc5-81c8-ec0b96a4647a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0e0e478-hmwl5\" (UID: \"9dc68081-80ec-4fc5-81c8-ec0b96a4647a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e0e478-hmwl5" Nov 03 14:28:40.838060 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:40.838019 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv876\" (UniqueName: \"kubernetes.io/projected/9dc68081-80ec-4fc5-81c8-ec0b96a4647a-kube-api-access-zv876\") pod \"maas-api-revoke-sub-b0e0e478-hmwl5\" (UID: \"9dc68081-80ec-4fc5-81c8-ec0b96a4647a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e0e478-hmwl5" Nov 03 14:28:40.885532 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:40.885510 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e0e478-hmwl5" Nov 03 14:28:41.202193 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:41.202161 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0e0e478-hmwl5"] Nov 03 14:28:41.203496 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:28:41.203467 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dc68081_80ec_4fc5_81c8_ec0b96a4647a.slice/crio-29b562fdd6389286a2c1a73862abb436768e01cff816f73bfeae3bf93a9e2c96 WatchSource:0}: Error finding container 29b562fdd6389286a2c1a73862abb436768e01cff816f73bfeae3bf93a9e2c96: Status 404 returned error can't find the container with id 29b562fdd6389286a2c1a73862abb436768e01cff816f73bfeae3bf93a9e2c96 Nov 03 14:28:42.142094 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:42.142058 5060 generic.go:356] "Generic (PLEG): container finished" podID="9dc68081-80ec-4fc5-81c8-ec0b96a4647a" containerID="376627bb6ce269d70f43c84e2ae1a64ddb804da5fe8100ee4c608cd7a5c9d1a1" exitCode=0 Nov 03 14:28:42.142486 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:42.142124 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e0e478-hmwl5" event={"ID":"9dc68081-80ec-4fc5-81c8-ec0b96a4647a","Type":"ContainerDied","Data":"376627bb6ce269d70f43c84e2ae1a64ddb804da5fe8100ee4c608cd7a5c9d1a1"} Nov 03 14:28:42.142486 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:42.142171 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e0e478-hmwl5" event={"ID":"9dc68081-80ec-4fc5-81c8-ec0b96a4647a","Type":"ContainerStarted","Data":"29b562fdd6389286a2c1a73862abb436768e01cff816f73bfeae3bf93a9e2c96"} Nov 03 14:28:43.304785 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:43.304754 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e0e478-hmwl5" Nov 03 14:28:43.446926 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:43.446847 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dc68081-80ec-4fc5-81c8-ec0b96a4647a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dc68081-80ec-4fc5-81c8-ec0b96a4647a-maas-api-xks-ca\") pod \"9dc68081-80ec-4fc5-81c8-ec0b96a4647a\" (UID: \"9dc68081-80ec-4fc5-81c8-ec0b96a4647a\") " Nov 03 14:28:43.447093 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:43.446982 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dc68081-80ec-4fc5-81c8-ec0b96a4647a-kube-api-access-zv876\" (UniqueName: \"kubernetes.io/projected/9dc68081-80ec-4fc5-81c8-ec0b96a4647a-kube-api-access-zv876\") pod \"9dc68081-80ec-4fc5-81c8-ec0b96a4647a\" (UID: \"9dc68081-80ec-4fc5-81c8-ec0b96a4647a\") " Nov 03 14:28:43.447093 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:43.447052 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dc68081-80ec-4fc5-81c8-ec0b96a4647a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dc68081-80ec-4fc5-81c8-ec0b96a4647a-maas-api-openshift-ca\") pod \"9dc68081-80ec-4fc5-81c8-ec0b96a4647a\" (UID: \"9dc68081-80ec-4fc5-81c8-ec0b96a4647a\") " Nov 03 14:28:43.447435 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:43.447405 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dc68081-80ec-4fc5-81c8-ec0b96a4647a-maas-api-openshift-ca" pod "9dc68081-80ec-4fc5-81c8-ec0b96a4647a" (UID: "9dc68081-80ec-4fc5-81c8-ec0b96a4647a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:28:43.449088 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:43.449064 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dc68081-80ec-4fc5-81c8-ec0b96a4647a-kube-api-access-zv876" pod "9dc68081-80ec-4fc5-81c8-ec0b96a4647a" (UID: "9dc68081-80ec-4fc5-81c8-ec0b96a4647a"). InnerVolumeSpecName "kube-api-access-zv876". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:28:43.449178 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:43.449089 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dc68081-80ec-4fc5-81c8-ec0b96a4647a-maas-api-xks-ca" pod "9dc68081-80ec-4fc5-81c8-ec0b96a4647a" (UID: "9dc68081-80ec-4fc5-81c8-ec0b96a4647a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:28:43.548763 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:43.548724 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dc68081-80ec-4fc5-81c8-ec0b96a4647a-maas-api-openshift-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:28:43.548763 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:43.548762 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dc68081-80ec-4fc5-81c8-ec0b96a4647a-maas-api-xks-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:28:43.549053 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:43.548776 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv876\" (UniqueName: \"kubernetes.io/projected/9dc68081-80ec-4fc5-81c8-ec0b96a4647a-kube-api-access-zv876\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:28:44.158591 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:44.158551 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e0e478-hmwl5" event={"ID":"9dc68081-80ec-4fc5-81c8-ec0b96a4647a","Type":"ContainerDied","Data":"29b562fdd6389286a2c1a73862abb436768e01cff816f73bfeae3bf93a9e2c96"} Nov 03 14:28:44.158591 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:44.158581 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e0e478-hmwl5" Nov 03 14:28:44.158591 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:44.158598 5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29b562fdd6389286a2c1a73862abb436768e01cff816f73bfeae3bf93a9e2c96" Nov 03 14:28:48.608993 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:48.608944 5060 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:28:48.609790 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:48.609749 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" podUID="b3327f34-66f3-4a70-915a-aa5ad8685fad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:28:51.026637 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:51.026599 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-957aebcd-dpbfp"] Nov 03 14:28:51.027168 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:51.027153 5060 cpu_manager.go:414] "containerMap: removing container" podUID="9dc68081-80ec-4fc5-81c8-ec0b96a4647a" containerName="revoke-keys" Nov 03 14:28:51.027210 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:51.027170 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dc68081-80ec-4fc5-81c8-ec0b96a4647a" containerName="revoke-keys" Nov 03 14:28:51.027283 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:51.027274 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dc68081-80ec-4fc5-81c8-ec0b96a4647a" containerName="revoke-keys" Nov 03 14:28:51.031001 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:51.030984 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-957aebcd-dpbfp" Nov 03 14:28:51.053412 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:51.053390 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-957aebcd-dpbfp"] Nov 03 14:28:51.062067 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:51.062041 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:28:51.062204 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:51.062140 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:28:51.062204 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:51.062179 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\"" Nov 03 14:28:51.062360 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:51.062340 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:28:51.111253 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:51.111205 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0877646-7ea1-4320-b6de-888cfd2fdce7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-957aebcd-dpbfp\" (UID: \"e0877646-7ea1-4320-b6de-888cfd2fdce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-957aebcd-dpbfp" Nov 03 14:28:51.111253 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:51.111261 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0877646-7ea1-4320-b6de-888cfd2fdce7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-957aebcd-dpbfp\" (UID: \"e0877646-7ea1-4320-b6de-888cfd2fdce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-957aebcd-dpbfp" Nov 03 14:28:51.111457 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:51.111434 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-878w4\" (UniqueName: \"kubernetes.io/projected/e0877646-7ea1-4320-b6de-888cfd2fdce7-kube-api-access-878w4\") pod \"maas-api-revoke-sub-957aebcd-dpbfp\" (UID: \"e0877646-7ea1-4320-b6de-888cfd2fdce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-957aebcd-dpbfp" Nov 03 14:28:51.212420 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:51.212378 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0877646-7ea1-4320-b6de-888cfd2fdce7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-957aebcd-dpbfp\" (UID: \"e0877646-7ea1-4320-b6de-888cfd2fdce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-957aebcd-dpbfp" Nov 03 14:28:51.212614 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:51.212425 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0877646-7ea1-4320-b6de-888cfd2fdce7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-957aebcd-dpbfp\" (UID: \"e0877646-7ea1-4320-b6de-888cfd2fdce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-957aebcd-dpbfp" Nov 03 14:28:51.212614 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:51.212536 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-878w4\" (UniqueName: \"kubernetes.io/projected/e0877646-7ea1-4320-b6de-888cfd2fdce7-kube-api-access-878w4\") pod \"maas-api-revoke-sub-957aebcd-dpbfp\" (UID: \"e0877646-7ea1-4320-b6de-888cfd2fdce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-957aebcd-dpbfp" Nov 03 14:28:51.213069 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:51.213047 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0877646-7ea1-4320-b6de-888cfd2fdce7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-957aebcd-dpbfp\" (UID: \"e0877646-7ea1-4320-b6de-888cfd2fdce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-957aebcd-dpbfp" Nov 03 14:28:51.214801 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:51.214778 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0877646-7ea1-4320-b6de-888cfd2fdce7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-957aebcd-dpbfp\" (UID: \"e0877646-7ea1-4320-b6de-888cfd2fdce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-957aebcd-dpbfp" Nov 03 14:28:51.235824 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:51.235789 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-878w4\" (UniqueName: \"kubernetes.io/projected/e0877646-7ea1-4320-b6de-888cfd2fdce7-kube-api-access-878w4\") pod \"maas-api-revoke-sub-957aebcd-dpbfp\" (UID: \"e0877646-7ea1-4320-b6de-888cfd2fdce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-957aebcd-dpbfp" Nov 03 14:28:51.344936 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:51.344854 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-957aebcd-dpbfp" Nov 03 14:28:51.659520 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:28:51.659485 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0877646_7ea1_4320_b6de_888cfd2fdce7.slice/crio-90ba7716c025d3d6c4d03e542fcab67abe6a390955b63136159f216ccf5dc975 WatchSource:0}: Error finding container 90ba7716c025d3d6c4d03e542fcab67abe6a390955b63136159f216ccf5dc975: Status 404 returned error can't find the container with id 90ba7716c025d3d6c4d03e542fcab67abe6a390955b63136159f216ccf5dc975 Nov 03 14:28:51.659806 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:51.659779 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-957aebcd-dpbfp"] Nov 03 14:28:52.235823 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:52.235749 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-957aebcd-dpbfp" event={"ID":"e0877646-7ea1-4320-b6de-888cfd2fdce7","Type":"ContainerStarted","Data":"4cd0350e5d34c0c87de155c5e498a4b4b04040c21f7b3f7f3b201aa6a97c3b7a"} Nov 03 14:28:52.235823 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:52.235798 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-957aebcd-dpbfp" event={"ID":"e0877646-7ea1-4320-b6de-888cfd2fdce7","Type":"ContainerStarted","Data":"90ba7716c025d3d6c4d03e542fcab67abe6a390955b63136159f216ccf5dc975"} Nov 03 14:28:52.282223 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:52.282170 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-957aebcd-dpbfp" podStartSLOduration=2.282158335 podStartE2EDuration="2.282158335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:28:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:28:52.278193127 +0000 UTC m=+1066.166982720" watchObservedRunningTime="2026-11-03 14:28:52.282158335 +0000 UTC m=+1066.170947908" Nov 03 14:28:53.244636 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:53.244599 5060 generic.go:356] "Generic (PLEG): container finished" podID="e0877646-7ea1-4320-b6de-888cfd2fdce7" containerID="4cd0350e5d34c0c87de155c5e498a4b4b04040c21f7b3f7f3b201aa6a97c3b7a" exitCode=0 Nov 03 14:28:53.245179 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:53.244683 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-957aebcd-dpbfp" event={"ID":"e0877646-7ea1-4320-b6de-888cfd2fdce7","Type":"ContainerDied","Data":"4cd0350e5d34c0c87de155c5e498a4b4b04040c21f7b3f7f3b201aa6a97c3b7a"} Nov 03 14:28:54.413503 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:54.413480 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-957aebcd-dpbfp" Nov 03 14:28:54.553664 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:54.553572 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0877646-7ea1-4320-b6de-888cfd2fdce7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0877646-7ea1-4320-b6de-888cfd2fdce7-maas-api-openshift-ca\") pod \"e0877646-7ea1-4320-b6de-888cfd2fdce7\" (UID: \"e0877646-7ea1-4320-b6de-888cfd2fdce7\") " Nov 03 14:28:54.553850 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:54.553710 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0877646-7ea1-4320-b6de-888cfd2fdce7-kube-api-access-878w4\" (UniqueName: \"kubernetes.io/projected/e0877646-7ea1-4320-b6de-888cfd2fdce7-kube-api-access-878w4\") pod \"e0877646-7ea1-4320-b6de-888cfd2fdce7\" (UID: \"e0877646-7ea1-4320-b6de-888cfd2fdce7\") " Nov 03 14:28:54.553929 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:54.553884 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0877646-7ea1-4320-b6de-888cfd2fdce7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0877646-7ea1-4320-b6de-888cfd2fdce7-maas-api-xks-ca\") pod \"e0877646-7ea1-4320-b6de-888cfd2fdce7\" (UID: \"e0877646-7ea1-4320-b6de-888cfd2fdce7\") " Nov 03 14:28:54.554011 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:54.553977 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0877646-7ea1-4320-b6de-888cfd2fdce7-maas-api-openshift-ca" pod "e0877646-7ea1-4320-b6de-888cfd2fdce7" (UID: "e0877646-7ea1-4320-b6de-888cfd2fdce7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:28:54.554272 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:54.554248 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0877646-7ea1-4320-b6de-888cfd2fdce7-maas-api-openshift-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:28:54.555975 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:54.555948 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0877646-7ea1-4320-b6de-888cfd2fdce7-maas-api-xks-ca" pod "e0877646-7ea1-4320-b6de-888cfd2fdce7" (UID: "e0877646-7ea1-4320-b6de-888cfd2fdce7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:28:54.556252 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:54.556234 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0877646-7ea1-4320-b6de-888cfd2fdce7-kube-api-access-878w4" pod "e0877646-7ea1-4320-b6de-888cfd2fdce7" (UID: "e0877646-7ea1-4320-b6de-888cfd2fdce7"). InnerVolumeSpecName "kube-api-access-878w4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:28:54.655399 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:54.655352 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0877646-7ea1-4320-b6de-888cfd2fdce7-maas-api-xks-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:28:54.655399 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:54.655398 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-878w4\" (UniqueName: \"kubernetes.io/projected/e0877646-7ea1-4320-b6de-888cfd2fdce7-kube-api-access-878w4\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:28:55.262639 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:55.262610 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-957aebcd-dpbfp" Nov 03 14:28:55.262901 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:55.262653 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-957aebcd-dpbfp" event={"ID":"e0877646-7ea1-4320-b6de-888cfd2fdce7","Type":"ContainerDied","Data":"90ba7716c025d3d6c4d03e542fcab67abe6a390955b63136159f216ccf5dc975"} Nov 03 14:28:55.262901 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:55.262697 5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90ba7716c025d3d6c4d03e542fcab67abe6a390955b63136159f216ccf5dc975" Nov 03 14:28:57.248329 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:57.248292 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-238b8528-spcqq"] Nov 03 14:28:57.248822 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:57.248807 5060 cpu_manager.go:414] "containerMap: removing container" podUID="e0877646-7ea1-4320-b6de-888cfd2fdce7" containerName="revoke-keys" Nov 03 14:28:57.248884 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:57.248823 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0877646-7ea1-4320-b6de-888cfd2fdce7" containerName="revoke-keys" Nov 03 14:28:57.248934 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:57.248924 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0877646-7ea1-4320-b6de-888cfd2fdce7" containerName="revoke-keys" Nov 03 14:28:57.252291 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:57.252264 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-238b8528-spcqq" Nov 03 14:28:57.279961 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:57.279001 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-238b8528-spcqq"] Nov 03 14:28:57.287195 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:57.287170 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\"" Nov 03 14:28:57.287355 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:57.287238 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:28:57.287355 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:57.287291 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:28:57.287461 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:57.287379 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:28:57.311575 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:57.311540 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6bd7\" (UniqueName: \"kubernetes.io/projected/fcfd1798-5f57-4e1c-b57e-8f4101883025-kube-api-access-z6bd7\") pod \"maas-api-revoke-sub-238b8528-spcqq\" (UID: \"fcfd1798-5f57-4e1c-b57e-8f4101883025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-238b8528-spcqq" Nov 03 14:28:57.311871 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:57.311848 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcfd1798-5f57-4e1c-b57e-8f4101883025-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-238b8528-spcqq\" (UID: \"fcfd1798-5f57-4e1c-b57e-8f4101883025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-238b8528-spcqq" Nov 03 14:28:57.312066 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:57.312041 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcfd1798-5f57-4e1c-b57e-8f4101883025-maas-api-xks-ca\") pod \"maas-api-revoke-sub-238b8528-spcqq\" (UID: \"fcfd1798-5f57-4e1c-b57e-8f4101883025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-238b8528-spcqq" Nov 03 14:28:57.413859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:57.413802 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6bd7\" (UniqueName: \"kubernetes.io/projected/fcfd1798-5f57-4e1c-b57e-8f4101883025-kube-api-access-z6bd7\") pod \"maas-api-revoke-sub-238b8528-spcqq\" (UID: \"fcfd1798-5f57-4e1c-b57e-8f4101883025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-238b8528-spcqq" Nov 03 14:28:57.414068 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:57.413881 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcfd1798-5f57-4e1c-b57e-8f4101883025-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-238b8528-spcqq\" (UID: \"fcfd1798-5f57-4e1c-b57e-8f4101883025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-238b8528-spcqq" Nov 03 14:28:57.414068 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:57.413927 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcfd1798-5f57-4e1c-b57e-8f4101883025-maas-api-xks-ca\") pod \"maas-api-revoke-sub-238b8528-spcqq\" (UID: \"fcfd1798-5f57-4e1c-b57e-8f4101883025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-238b8528-spcqq" Nov 03 14:28:57.414740 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:57.414705 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcfd1798-5f57-4e1c-b57e-8f4101883025-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-238b8528-spcqq\" (UID: \"fcfd1798-5f57-4e1c-b57e-8f4101883025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-238b8528-spcqq" Nov 03 14:28:57.416330 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:57.416299 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcfd1798-5f57-4e1c-b57e-8f4101883025-maas-api-xks-ca\") pod \"maas-api-revoke-sub-238b8528-spcqq\" (UID: \"fcfd1798-5f57-4e1c-b57e-8f4101883025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-238b8528-spcqq" Nov 03 14:28:57.440497 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:57.440466 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6bd7\" (UniqueName: \"kubernetes.io/projected/fcfd1798-5f57-4e1c-b57e-8f4101883025-kube-api-access-z6bd7\") pod \"maas-api-revoke-sub-238b8528-spcqq\" (UID: \"fcfd1798-5f57-4e1c-b57e-8f4101883025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-238b8528-spcqq" Nov 03 14:28:57.565828 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:57.565755 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-238b8528-spcqq" Nov 03 14:28:57.883862 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:57.883808 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-238b8528-spcqq"] Nov 03 14:28:57.885528 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:28:57.885495 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcfd1798_5f57_4e1c_b57e_8f4101883025.slice/crio-8bac305afff66bd864d728c7662f5c6ec9f9a3b1893c22db7cb1a22e24c58812 WatchSource:0}: Error finding container 8bac305afff66bd864d728c7662f5c6ec9f9a3b1893c22db7cb1a22e24c58812: Status 404 returned error can't find the container with id 8bac305afff66bd864d728c7662f5c6ec9f9a3b1893c22db7cb1a22e24c58812 Nov 03 14:28:58.284541 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:58.284506 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-238b8528-spcqq" event={"ID":"fcfd1798-5f57-4e1c-b57e-8f4101883025","Type":"ContainerStarted","Data":"7e4c75cabe8c013afec09828f0f90716396847464584b526b13232e723c3714a"} Nov 03 14:28:58.284952 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:58.284561 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-238b8528-spcqq" event={"ID":"fcfd1798-5f57-4e1c-b57e-8f4101883025","Type":"ContainerStarted","Data":"8bac305afff66bd864d728c7662f5c6ec9f9a3b1893c22db7cb1a22e24c58812"} Nov 03 14:28:58.331438 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:58.331385 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-238b8528-spcqq" podStartSLOduration=1.331372115 podStartE2EDuration="1.331372115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:28:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:28:58.330849267 +0000 UTC m=+1072.219638842" watchObservedRunningTime="2026-11-03 14:28:58.331372115 +0000 UTC m=+1072.220161695" Nov 03 14:28:58.809193 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:58.809156 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-shp6t-59fn9"] Nov 03 14:28:58.813140 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:58.813116 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-shp6t-59fn9" Nov 03 14:28:58.836338 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:58.836301 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-shp6t-59fn9"] Nov 03 14:28:58.844887 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:58.844859 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-qgp8d\"" Nov 03 14:28:58.929250 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:58.929219 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5skb\" (UniqueName: \"kubernetes.io/projected/baf0a8cf-ee1d-4897-9b75-d28b2023c299-kube-api-access-l5skb\") pod \"periodic-gathering-shp6t-59fn9\" (UID: \"baf0a8cf-ee1d-4897-9b75-d28b2023c299\") " pod="openshift-insights/periodic-gathering-shp6t-59fn9" Nov 03 14:28:58.929250 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:58.929261 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/baf0a8cf-ee1d-4897-9b75-d28b2023c299-archives-path\") pod \"periodic-gathering-shp6t-59fn9\" (UID: \"baf0a8cf-ee1d-4897-9b75-d28b2023c299\") " pod="openshift-insights/periodic-gathering-shp6t-59fn9" Nov 03 14:28:58.929496 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:58.929374 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/baf0a8cf-ee1d-4897-9b75-d28b2023c299-trusted-ca-bundle\") pod \"periodic-gathering-shp6t-59fn9\" (UID: \"baf0a8cf-ee1d-4897-9b75-d28b2023c299\") " pod="openshift-insights/periodic-gathering-shp6t-59fn9" Nov 03 14:28:58.929545 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:58.929516 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/baf0a8cf-ee1d-4897-9b75-d28b2023c299-service-ca-bundle\") pod \"periodic-gathering-shp6t-59fn9\" (UID: \"baf0a8cf-ee1d-4897-9b75-d28b2023c299\") " pod="openshift-insights/periodic-gathering-shp6t-59fn9" Nov 03 14:28:59.030888 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:59.030826 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5skb\" (UniqueName: \"kubernetes.io/projected/baf0a8cf-ee1d-4897-9b75-d28b2023c299-kube-api-access-l5skb\") pod \"periodic-gathering-shp6t-59fn9\" (UID: \"baf0a8cf-ee1d-4897-9b75-d28b2023c299\") " pod="openshift-insights/periodic-gathering-shp6t-59fn9" Nov 03 14:28:59.031083 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:59.030907 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/baf0a8cf-ee1d-4897-9b75-d28b2023c299-archives-path\") pod \"periodic-gathering-shp6t-59fn9\" (UID: \"baf0a8cf-ee1d-4897-9b75-d28b2023c299\") " pod="openshift-insights/periodic-gathering-shp6t-59fn9" Nov 03 14:28:59.031083 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:59.030961 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/baf0a8cf-ee1d-4897-9b75-d28b2023c299-trusted-ca-bundle\") pod \"periodic-gathering-shp6t-59fn9\" (UID: \"baf0a8cf-ee1d-4897-9b75-d28b2023c299\") " pod="openshift-insights/periodic-gathering-shp6t-59fn9" Nov 03 14:28:59.031216 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:59.031099 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/baf0a8cf-ee1d-4897-9b75-d28b2023c299-service-ca-bundle\") pod \"periodic-gathering-shp6t-59fn9\" (UID: \"baf0a8cf-ee1d-4897-9b75-d28b2023c299\") " pod="openshift-insights/periodic-gathering-shp6t-59fn9" Nov 03 14:28:59.031418 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:59.031389 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/baf0a8cf-ee1d-4897-9b75-d28b2023c299-archives-path\") pod \"periodic-gathering-shp6t-59fn9\" (UID: \"baf0a8cf-ee1d-4897-9b75-d28b2023c299\") " pod="openshift-insights/periodic-gathering-shp6t-59fn9" Nov 03 14:28:59.031703 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:59.031682 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/baf0a8cf-ee1d-4897-9b75-d28b2023c299-service-ca-bundle\") pod \"periodic-gathering-shp6t-59fn9\" (UID: \"baf0a8cf-ee1d-4897-9b75-d28b2023c299\") " pod="openshift-insights/periodic-gathering-shp6t-59fn9" Nov 03 14:28:59.031817 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:59.031795 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/baf0a8cf-ee1d-4897-9b75-d28b2023c299-trusted-ca-bundle\") pod \"periodic-gathering-shp6t-59fn9\" (UID: \"baf0a8cf-ee1d-4897-9b75-d28b2023c299\") " pod="openshift-insights/periodic-gathering-shp6t-59fn9" Nov 03 14:28:59.053028 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:59.053001 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5skb\" (UniqueName: \"kubernetes.io/projected/baf0a8cf-ee1d-4897-9b75-d28b2023c299-kube-api-access-l5skb\") pod \"periodic-gathering-shp6t-59fn9\" (UID: \"baf0a8cf-ee1d-4897-9b75-d28b2023c299\") " pod="openshift-insights/periodic-gathering-shp6t-59fn9" Nov 03 14:28:59.161021 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:59.160988 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-shp6t-59fn9" Nov 03 14:28:59.292607 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:59.292521 5060 generic.go:356] "Generic (PLEG): container finished" podID="fcfd1798-5f57-4e1c-b57e-8f4101883025" containerID="7e4c75cabe8c013afec09828f0f90716396847464584b526b13232e723c3714a" exitCode=0 Nov 03 14:28:59.293036 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:59.292609 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-238b8528-spcqq" event={"ID":"fcfd1798-5f57-4e1c-b57e-8f4101883025","Type":"ContainerDied","Data":"7e4c75cabe8c013afec09828f0f90716396847464584b526b13232e723c3714a"} Nov 03 14:28:59.471142 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:28:59.471115 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-shp6t-59fn9"] Nov 03 14:28:59.471952 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:28:59.471919 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaf0a8cf_ee1d_4897_9b75_d28b2023c299.slice/crio-1e413ce5274345f3fd85e3ab04c27940caff6cd5815b24abdc2acf2871f5d438 WatchSource:0}: Error finding container 1e413ce5274345f3fd85e3ab04c27940caff6cd5815b24abdc2acf2871f5d438: Status 404 returned error can't find the container with id 1e413ce5274345f3fd85e3ab04c27940caff6cd5815b24abdc2acf2871f5d438 Nov 03 14:29:00.301074 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:00.301039 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-shp6t-59fn9" event={"ID":"baf0a8cf-ee1d-4897-9b75-d28b2023c299","Type":"ContainerStarted","Data":"b38488471741ee7d8d82c8213d498089d51d8e7cb0fbbca56d38d46eb760f369"} Nov 03 14:29:00.301505 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:00.301088 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-shp6t-59fn9" event={"ID":"baf0a8cf-ee1d-4897-9b75-d28b2023c299","Type":"ContainerStarted","Data":"1e413ce5274345f3fd85e3ab04c27940caff6cd5815b24abdc2acf2871f5d438"} Nov 03 14:29:00.351854 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:00.351793 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-shp6t-59fn9" podStartSLOduration=2.351778891 podStartE2EDuration="2.351778891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:29:00.349323255 +0000 UTC m=+1074.238112836" watchObservedRunningTime="2026-11-03 14:29:00.351778891 +0000 UTC m=+1074.240568469" Nov 03 14:29:00.467734 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:00.467706 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-238b8528-spcqq" Nov 03 14:29:00.559377 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:00.559260 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcfd1798-5f57-4e1c-b57e-8f4101883025-kube-api-access-z6bd7\" (UniqueName: \"kubernetes.io/projected/fcfd1798-5f57-4e1c-b57e-8f4101883025-kube-api-access-z6bd7\") pod \"fcfd1798-5f57-4e1c-b57e-8f4101883025\" (UID: \"fcfd1798-5f57-4e1c-b57e-8f4101883025\") " Nov 03 14:29:00.559540 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:00.559400 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcfd1798-5f57-4e1c-b57e-8f4101883025-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcfd1798-5f57-4e1c-b57e-8f4101883025-maas-api-xks-ca\") pod \"fcfd1798-5f57-4e1c-b57e-8f4101883025\" (UID: \"fcfd1798-5f57-4e1c-b57e-8f4101883025\") " Nov 03 14:29:00.559540 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:00.559486 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcfd1798-5f57-4e1c-b57e-8f4101883025-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcfd1798-5f57-4e1c-b57e-8f4101883025-maas-api-openshift-ca\") pod \"fcfd1798-5f57-4e1c-b57e-8f4101883025\" (UID: \"fcfd1798-5f57-4e1c-b57e-8f4101883025\") " Nov 03 14:29:00.583193 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:00.583154 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcfd1798-5f57-4e1c-b57e-8f4101883025-maas-api-openshift-ca" pod "fcfd1798-5f57-4e1c-b57e-8f4101883025" (UID: "fcfd1798-5f57-4e1c-b57e-8f4101883025"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:29:00.642715 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:00.642658 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcfd1798-5f57-4e1c-b57e-8f4101883025-maas-api-xks-ca" pod "fcfd1798-5f57-4e1c-b57e-8f4101883025" (UID: "fcfd1798-5f57-4e1c-b57e-8f4101883025"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:29:00.643110 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:00.643091 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcfd1798-5f57-4e1c-b57e-8f4101883025-kube-api-access-z6bd7" pod "fcfd1798-5f57-4e1c-b57e-8f4101883025" (UID: "fcfd1798-5f57-4e1c-b57e-8f4101883025"). InnerVolumeSpecName "kube-api-access-z6bd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:29:00.661022 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:00.660983 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcfd1798-5f57-4e1c-b57e-8f4101883025-maas-api-openshift-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:29:00.661022 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:00.661021 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6bd7\" (UniqueName: \"kubernetes.io/projected/fcfd1798-5f57-4e1c-b57e-8f4101883025-kube-api-access-z6bd7\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:29:00.661262 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:00.661037 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcfd1798-5f57-4e1c-b57e-8f4101883025-maas-api-xks-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:29:01.318855 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:01.318307 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-238b8528-spcqq" Nov 03 14:29:01.331858 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:01.324886 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-238b8528-spcqq" event={"ID":"fcfd1798-5f57-4e1c-b57e-8f4101883025","Type":"ContainerDied","Data":"8bac305afff66bd864d728c7662f5c6ec9f9a3b1893c22db7cb1a22e24c58812"} Nov 03 14:29:01.331858 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:01.324951 5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bac305afff66bd864d728c7662f5c6ec9f9a3b1893c22db7cb1a22e24c58812" Nov 03 14:29:03.616242 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:03.616206 5060 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:29:03.616864 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:03.616814 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" podUID="b3327f34-66f3-4a70-915a-aa5ad8685fad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:29:03.846827 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:03.846772 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz" podUID="930bcc45-2ffe-4502-aff6-03761f25ee62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 14:29:04.056873 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:04.055985 5060 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:29:04.056873 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:04.056058 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" podUID="f859590f-c6cb-4b6c-bfd2-ebde80002e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:29:04.409916 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:04.409253 5060 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:29:04.409916 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:04.409313 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" podUID="b671512d-59b5-4ddd-959d-1d9150d41268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:29:04.517280 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:04.517228 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98" podUID="3aaaeea4-134f-4d17-a180-b4857c6425d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 03 14:29:04.782055 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:04.782013 5060 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:29:04.812856 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:04.789902 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" podUID="f6695b0e-c9b6-4119-a0b2-508b212a3ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:29:04.857313 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:04.857278 5060 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:29:04.857607 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:04.857572 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" podUID="a88d21ce-086a-414a-911a-5561e2cb8023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:29:06.544470 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:06.544436 5060 scope.go:122] "RemoveContainer" containerID="397f93e82ffb30e8f057a650c243eca5bdad33f39b2cb64a9406281149c8da15" Nov 03 14:29:07.006096 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:07.006051 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj"] Nov 03 14:29:07.006881 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:07.006858 5060 cpu_manager.go:414] "containerMap: removing container" podUID="fcfd1798-5f57-4e1c-b57e-8f4101883025" containerName="revoke-keys" Nov 03 14:29:07.006881 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:07.006882 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcfd1798-5f57-4e1c-b57e-8f4101883025" containerName="revoke-keys" Nov 03 14:29:07.007076 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:07.007063 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcfd1798-5f57-4e1c-b57e-8f4101883025" containerName="revoke-keys" Nov 03 14:29:07.012508 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:07.012484 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" Nov 03 14:29:07.024605 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:07.024579 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj"] Nov 03 14:29:07.032639 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:07.032611 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-4a67e25f-openshift-default-dockercfg-v66ww\"" Nov 03 14:29:07.180131 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:07.180045 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07fa990f-370e-4d84-b7e6-340f4caa6348-istio-data\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj\" (UID: \"07fa990f-370e-4d84-b7e6-340f4caa6348\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" Nov 03 14:29:07.180489 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:07.180462 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07fa990f-370e-4d84-b7e6-340f4caa6348-workload-certs\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj\" (UID: \"07fa990f-370e-4d84-b7e6-340f4caa6348\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" Nov 03 14:29:07.180662 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:07.180647 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07fa990f-370e-4d84-b7e6-340f4caa6348-istio-podinfo\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj\" (UID: \"07fa990f-370e-4d84-b7e6-340f4caa6348\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" Nov 03 14:29:07.180770 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:07.180753 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tblzp\" (UniqueName: \"kubernetes.io/projected/07fa990f-370e-4d84-b7e6-340f4caa6348-kube-api-access-tblzp\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj\" (UID: \"07fa990f-370e-4d84-b7e6-340f4caa6348\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" Nov 03 14:29:07.181125 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:07.180944 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07fa990f-370e-4d84-b7e6-340f4caa6348-workload-socket\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj\" (UID: \"07fa990f-370e-4d84-b7e6-340f4caa6348\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" Nov 03 14:29:07.181331 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:07.181312 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07fa990f-370e-4d84-b7e6-340f4caa6348-istio-envoy\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj\" (UID: \"07fa990f-370e-4d84-b7e6-340f4caa6348\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" Nov 03 14:29:07.182053 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:07.182003 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07fa990f-370e-4d84-b7e6-340f4caa6348-istiod-ca-cert\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj\" (UID: \"07fa990f-370e-4d84-b7e6-340f4caa6348\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" Nov 03 14:29:07.182404 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:07.182360 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07fa990f-370e-4d84-b7e6-340f4caa6348-istio-token\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj\" (UID: \"07fa990f-370e-4d84-b7e6-340f4caa6348\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" Nov 03 14:29:07.182600 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:07.182582 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07fa990f-370e-4d84-b7e6-340f4caa6348-credential-socket\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj\" (UID: \"07fa990f-370e-4d84-b7e6-340f4caa6348\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" Nov 03 14:29:07.293033 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:07.283773 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07fa990f-370e-4d84-b7e6-340f4caa6348-workload-socket\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj\" (UID: \"07fa990f-370e-4d84-b7e6-340f4caa6348\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" Nov 03 14:29:07.293033 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:07.283883 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07fa990f-370e-4d84-b7e6-340f4caa6348-istio-envoy\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj\" (UID: \"07fa990f-370e-4d84-b7e6-340f4caa6348\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" Nov 03 14:29:07.293033 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:07.283936 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07fa990f-370e-4d84-b7e6-340f4caa6348-istiod-ca-cert\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj\" (UID: \"07fa990f-370e-4d84-b7e6-340f4caa6348\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" Nov 03 14:29:07.293033 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:07.283974 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07fa990f-370e-4d84-b7e6-340f4caa6348-istio-token\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj\" (UID: \"07fa990f-370e-4d84-b7e6-340f4caa6348\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" Nov 03 14:29:07.293033 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:07.284019 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07fa990f-370e-4d84-b7e6-340f4caa6348-credential-socket\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj\" (UID: \"07fa990f-370e-4d84-b7e6-340f4caa6348\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" Nov 03 14:29:07.293033 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:07.284079 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07fa990f-370e-4d84-b7e6-340f4caa6348-istio-data\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj\" (UID: \"07fa990f-370e-4d84-b7e6-340f4caa6348\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" Nov 03 14:29:07.293033 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:07.284116 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07fa990f-370e-4d84-b7e6-340f4caa6348-workload-certs\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj\" (UID: \"07fa990f-370e-4d84-b7e6-340f4caa6348\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" Nov 03 14:29:07.293033 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:07.284187 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07fa990f-370e-4d84-b7e6-340f4caa6348-istio-podinfo\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj\" (UID: \"07fa990f-370e-4d84-b7e6-340f4caa6348\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" Nov 03 14:29:07.293033 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:07.284213 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tblzp\" (UniqueName: \"kubernetes.io/projected/07fa990f-370e-4d84-b7e6-340f4caa6348-kube-api-access-tblzp\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj\" (UID: \"07fa990f-370e-4d84-b7e6-340f4caa6348\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" Nov 03 14:29:07.293033 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:07.285095 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07fa990f-370e-4d84-b7e6-340f4caa6348-workload-socket\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj\" (UID: \"07fa990f-370e-4d84-b7e6-340f4caa6348\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" Nov 03 14:29:07.293033 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:07.285954 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07fa990f-370e-4d84-b7e6-340f4caa6348-credential-socket\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj\" (UID: \"07fa990f-370e-4d84-b7e6-340f4caa6348\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" Nov 03 14:29:07.293033 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:07.285992 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07fa990f-370e-4d84-b7e6-340f4caa6348-workload-certs\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj\" (UID: \"07fa990f-370e-4d84-b7e6-340f4caa6348\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" Nov 03 14:29:07.293033 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:07.286276 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07fa990f-370e-4d84-b7e6-340f4caa6348-istiod-ca-cert\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj\" (UID: \"07fa990f-370e-4d84-b7e6-340f4caa6348\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" Nov 03 14:29:07.293033 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:07.287129 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07fa990f-370e-4d84-b7e6-340f4caa6348-istio-data\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj\" (UID: \"07fa990f-370e-4d84-b7e6-340f4caa6348\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" Nov 03 14:29:07.293033 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:07.288569 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07fa990f-370e-4d84-b7e6-340f4caa6348-istio-envoy\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj\" (UID: \"07fa990f-370e-4d84-b7e6-340f4caa6348\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" Nov 03 14:29:07.293033 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:07.290487 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07fa990f-370e-4d84-b7e6-340f4caa6348-istio-podinfo\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj\" (UID: \"07fa990f-370e-4d84-b7e6-340f4caa6348\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" Nov 03 14:29:07.308726 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:07.308687 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tblzp\" (UniqueName: \"kubernetes.io/projected/07fa990f-370e-4d84-b7e6-340f4caa6348-kube-api-access-tblzp\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj\" (UID: \"07fa990f-370e-4d84-b7e6-340f4caa6348\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" Nov 03 14:29:07.309451 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:07.309425 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07fa990f-370e-4d84-b7e6-340f4caa6348-istio-token\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj\" (UID: \"07fa990f-370e-4d84-b7e6-340f4caa6348\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" Nov 03 14:29:07.329378 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:07.329340 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" Nov 03 14:29:07.655580 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:07.655501 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj"] Nov 03 14:29:07.659469 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:29:07.659440 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07fa990f_370e_4d84_b7e6_340f4caa6348.slice/crio-81b596ceb1af600a150dafabe893671fdc0268e6479df3c014d90a2e3139aa86 WatchSource:0}: Error finding container 81b596ceb1af600a150dafabe893671fdc0268e6479df3c014d90a2e3139aa86: Status 404 returned error can't find the container with id 81b596ceb1af600a150dafabe893671fdc0268e6479df3c014d90a2e3139aa86 Nov 03 14:29:07.661948 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:07.661916 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:29:07.662034 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:07.661981 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:29:07.662034 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:07.662017 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:29:07.766017 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:07.765975 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" event={"ID":"07fa990f-370e-4d84-b7e6-340f4caa6348","Type":"ContainerStarted","Data":"f16df8daebf0a320416c241bacc909170a392e9e7159285076f80439fc860ac9"} Nov 03 14:29:07.766226 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:07.766042 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" event={"ID":"07fa990f-370e-4d84-b7e6-340f4caa6348","Type":"ContainerStarted","Data":"81b596ceb1af600a150dafabe893671fdc0268e6479df3c014d90a2e3139aa86"} Nov 03 14:29:07.822807 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:07.822741 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" podStartSLOduration=1.822720597 podStartE2EDuration="1.822720597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:29:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:29:07.821081324 +0000 UTC m=+1081.709870908" watchObservedRunningTime="2026-11-03 14:29:07.822720597 +0000 UTC m=+1081.711510179" Nov 03 14:29:08.329475 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:08.329440 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" Nov 03 14:29:08.336011 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:08.335982 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" Nov 03 14:29:08.773432 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:08.773397 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" Nov 03 14:29:08.774450 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:08.774422 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" Nov 03 14:29:09.447920 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:29:09.447885 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:29:11.231009 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:11.230966 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbd38938-t48fd"] Nov 03 14:29:11.235176 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:11.235152 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbd38938-t48fd" Nov 03 14:29:11.257460 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:11.257421 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbd38938-t48fd"] Nov 03 14:29:11.266266 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:11.266241 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:29:11.266387 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:11.266296 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:29:11.273926 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:11.273901 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:29:11.274047 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:11.273988 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\"" Nov 03 14:29:11.334123 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:11.334090 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49301a5f-e1e4-4167-8bbb-311834b37475-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-cbd38938-t48fd\" (UID: \"49301a5f-e1e4-4167-8bbb-311834b37475\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbd38938-t48fd" Nov 03 14:29:11.334123 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:11.334131 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49301a5f-e1e4-4167-8bbb-311834b37475-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-cbd38938-t48fd\" (UID: \"49301a5f-e1e4-4167-8bbb-311834b37475\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbd38938-t48fd" Nov 03 14:29:11.334767 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:11.334740 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmvqs\" (UniqueName: \"kubernetes.io/projected/49301a5f-e1e4-4167-8bbb-311834b37475-kube-api-access-kmvqs\") pod \"maas-api-revoke-keys-e2e-ait-cbd38938-t48fd\" (UID: \"49301a5f-e1e4-4167-8bbb-311834b37475\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbd38938-t48fd" Nov 03 14:29:11.435354 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:11.435324 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49301a5f-e1e4-4167-8bbb-311834b37475-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-cbd38938-t48fd\" (UID: \"49301a5f-e1e4-4167-8bbb-311834b37475\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbd38938-t48fd" Nov 03 14:29:11.435525 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:11.435364 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49301a5f-e1e4-4167-8bbb-311834b37475-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-cbd38938-t48fd\" (UID: \"49301a5f-e1e4-4167-8bbb-311834b37475\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbd38938-t48fd" Nov 03 14:29:11.435613 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:11.435543 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmvqs\" (UniqueName: \"kubernetes.io/projected/49301a5f-e1e4-4167-8bbb-311834b37475-kube-api-access-kmvqs\") pod \"maas-api-revoke-keys-e2e-ait-cbd38938-t48fd\" (UID: \"49301a5f-e1e4-4167-8bbb-311834b37475\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbd38938-t48fd" Nov 03 14:29:11.436201 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:11.436169 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49301a5f-e1e4-4167-8bbb-311834b37475-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-cbd38938-t48fd\" (UID: \"49301a5f-e1e4-4167-8bbb-311834b37475\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbd38938-t48fd" Nov 03 14:29:11.437654 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:11.437629 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49301a5f-e1e4-4167-8bbb-311834b37475-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-cbd38938-t48fd\" (UID: \"49301a5f-e1e4-4167-8bbb-311834b37475\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbd38938-t48fd" Nov 03 14:29:11.457970 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:11.457947 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmvqs\" (UniqueName: \"kubernetes.io/projected/49301a5f-e1e4-4167-8bbb-311834b37475-kube-api-access-kmvqs\") pod \"maas-api-revoke-keys-e2e-ait-cbd38938-t48fd\" (UID: \"49301a5f-e1e4-4167-8bbb-311834b37475\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbd38938-t48fd" Nov 03 14:29:11.549989 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:11.549961 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbd38938-t48fd" Nov 03 14:29:11.858389 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:11.858349 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbd38938-t48fd"] Nov 03 14:29:11.859405 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:29:11.859377 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49301a5f_e1e4_4167_8bbb_311834b37475.slice/crio-04e845b1c53849e8ee4053def0abb461325665d9b5ba632dfc8629b4abbb5fed WatchSource:0}: Error finding container 04e845b1c53849e8ee4053def0abb461325665d9b5ba632dfc8629b4abbb5fed: Status 404 returned error can't find the container with id 04e845b1c53849e8ee4053def0abb461325665d9b5ba632dfc8629b4abbb5fed Nov 03 14:29:12.804242 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:12.804204 5060 generic.go:356] "Generic (PLEG): container finished" podID="49301a5f-e1e4-4167-8bbb-311834b37475" containerID="2d3f860a4d1e564b8e3dbb06f6a357a2f174ef50012cbd65dbe3cab55f750bf1" exitCode=0 Nov 03 14:29:12.804651 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:12.804287 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbd38938-t48fd" event={"ID":"49301a5f-e1e4-4167-8bbb-311834b37475","Type":"ContainerDied","Data":"2d3f860a4d1e564b8e3dbb06f6a357a2f174ef50012cbd65dbe3cab55f750bf1"} Nov 03 14:29:12.804651 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:12.804354 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbd38938-t48fd" event={"ID":"49301a5f-e1e4-4167-8bbb-311834b37475","Type":"ContainerStarted","Data":"04e845b1c53849e8ee4053def0abb461325665d9b5ba632dfc8629b4abbb5fed"} Nov 03 14:29:13.973079 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:13.973049 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbd38938-t48fd" Nov 03 14:29:14.076266 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:14.076183 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49301a5f-e1e4-4167-8bbb-311834b37475-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49301a5f-e1e4-4167-8bbb-311834b37475-maas-api-xks-ca\") pod \"49301a5f-e1e4-4167-8bbb-311834b37475\" (UID: \"49301a5f-e1e4-4167-8bbb-311834b37475\") " Nov 03 14:29:14.076266 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:14.076246 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49301a5f-e1e4-4167-8bbb-311834b37475-kube-api-access-kmvqs\" (UniqueName: \"kubernetes.io/projected/49301a5f-e1e4-4167-8bbb-311834b37475-kube-api-access-kmvqs\") pod \"49301a5f-e1e4-4167-8bbb-311834b37475\" (UID: \"49301a5f-e1e4-4167-8bbb-311834b37475\") " Nov 03 14:29:14.076487 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:14.076292 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49301a5f-e1e4-4167-8bbb-311834b37475-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49301a5f-e1e4-4167-8bbb-311834b37475-maas-api-openshift-ca\") pod \"49301a5f-e1e4-4167-8bbb-311834b37475\" (UID: \"49301a5f-e1e4-4167-8bbb-311834b37475\") " Nov 03 14:29:14.076789 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:14.076757 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49301a5f-e1e4-4167-8bbb-311834b37475-maas-api-openshift-ca" pod "49301a5f-e1e4-4167-8bbb-311834b37475" (UID: "49301a5f-e1e4-4167-8bbb-311834b37475"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:29:14.078505 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:14.078480 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49301a5f-e1e4-4167-8bbb-311834b37475-kube-api-access-kmvqs" pod "49301a5f-e1e4-4167-8bbb-311834b37475" (UID: "49301a5f-e1e4-4167-8bbb-311834b37475"). InnerVolumeSpecName "kube-api-access-kmvqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:29:14.078505 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:14.078494 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49301a5f-e1e4-4167-8bbb-311834b37475-maas-api-xks-ca" pod "49301a5f-e1e4-4167-8bbb-311834b37475" (UID: "49301a5f-e1e4-4167-8bbb-311834b37475"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:29:14.177849 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:14.177806 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49301a5f-e1e4-4167-8bbb-311834b37475-maas-api-xks-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:29:14.178010 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:14.177868 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmvqs\" (UniqueName: \"kubernetes.io/projected/49301a5f-e1e4-4167-8bbb-311834b37475-kube-api-access-kmvqs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:29:14.178010 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:14.177879 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49301a5f-e1e4-4167-8bbb-311834b37475-maas-api-openshift-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:29:14.855228 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:14.855184 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbd38938-t48fd" event={"ID":"49301a5f-e1e4-4167-8bbb-311834b37475","Type":"ContainerDied","Data":"04e845b1c53849e8ee4053def0abb461325665d9b5ba632dfc8629b4abbb5fed"} Nov 03 14:29:14.855535 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:14.855514 5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04e845b1c53849e8ee4053def0abb461325665d9b5ba632dfc8629b4abbb5fed" Nov 03 14:29:14.863886 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:14.855703 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbd38938-t48fd" Nov 03 14:29:17.272562 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:17.272528 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2"] Nov 03 14:29:17.273066 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:17.273052 5060 cpu_manager.go:414] "containerMap: removing container" podUID="49301a5f-e1e4-4167-8bbb-311834b37475" containerName="revoke-keys" Nov 03 14:29:17.273130 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:17.273067 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49301a5f-e1e4-4167-8bbb-311834b37475" containerName="revoke-keys" Nov 03 14:29:17.273172 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:17.273160 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="49301a5f-e1e4-4167-8bbb-311834b37475" containerName="revoke-keys" Nov 03 14:29:17.277055 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:17.277033 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" Nov 03 14:29:17.309366 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:17.309333 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2"] Nov 03 14:29:17.397369 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:17.397316 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/941df1f9-9c29-4205-b2fb-88167bd809ec-istiod-ca-cert\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2\" (UID: \"941df1f9-9c29-4205-b2fb-88167bd809ec\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" Nov 03 14:29:17.397369 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:17.397373 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/941df1f9-9c29-4205-b2fb-88167bd809ec-workload-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2\" (UID: \"941df1f9-9c29-4205-b2fb-88167bd809ec\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" Nov 03 14:29:17.397618 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:17.397486 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/941df1f9-9c29-4205-b2fb-88167bd809ec-istio-data\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2\" (UID: \"941df1f9-9c29-4205-b2fb-88167bd809ec\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" Nov 03 14:29:17.397618 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:17.397515 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/941df1f9-9c29-4205-b2fb-88167bd809ec-istio-envoy\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2\" (UID: \"941df1f9-9c29-4205-b2fb-88167bd809ec\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" Nov 03 14:29:17.397729 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:17.397654 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/941df1f9-9c29-4205-b2fb-88167bd809ec-credential-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2\" (UID: \"941df1f9-9c29-4205-b2fb-88167bd809ec\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" Nov 03 14:29:17.397810 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:17.397790 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/941df1f9-9c29-4205-b2fb-88167bd809ec-workload-certs\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2\" (UID: \"941df1f9-9c29-4205-b2fb-88167bd809ec\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" Nov 03 14:29:17.397894 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:17.397878 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzbj9\" (UniqueName: \"kubernetes.io/projected/941df1f9-9c29-4205-b2fb-88167bd809ec-kube-api-access-nzbj9\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2\" (UID: \"941df1f9-9c29-4205-b2fb-88167bd809ec\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" Nov 03 14:29:17.397954 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:17.397915 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/941df1f9-9c29-4205-b2fb-88167bd809ec-istio-token\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2\" (UID: \"941df1f9-9c29-4205-b2fb-88167bd809ec\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" Nov 03 14:29:17.398024 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:17.398010 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/941df1f9-9c29-4205-b2fb-88167bd809ec-istio-podinfo\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2\" (UID: \"941df1f9-9c29-4205-b2fb-88167bd809ec\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" Nov 03 14:29:17.498847 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:17.498806 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/941df1f9-9c29-4205-b2fb-88167bd809ec-istio-data\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2\" (UID: \"941df1f9-9c29-4205-b2fb-88167bd809ec\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" Nov 03 14:29:17.499071 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:17.498866 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/941df1f9-9c29-4205-b2fb-88167bd809ec-istio-envoy\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2\" (UID: \"941df1f9-9c29-4205-b2fb-88167bd809ec\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" Nov 03 14:29:17.499071 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:17.498939 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/941df1f9-9c29-4205-b2fb-88167bd809ec-credential-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2\" (UID: \"941df1f9-9c29-4205-b2fb-88167bd809ec\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" Nov 03 14:29:17.499071 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:17.499059 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/941df1f9-9c29-4205-b2fb-88167bd809ec-workload-certs\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2\" (UID: \"941df1f9-9c29-4205-b2fb-88167bd809ec\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" Nov 03 14:29:17.499238 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:17.499095 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzbj9\" (UniqueName: \"kubernetes.io/projected/941df1f9-9c29-4205-b2fb-88167bd809ec-kube-api-access-nzbj9\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2\" (UID: \"941df1f9-9c29-4205-b2fb-88167bd809ec\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" Nov 03 14:29:17.499238 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:17.499126 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/941df1f9-9c29-4205-b2fb-88167bd809ec-istio-token\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2\" (UID: \"941df1f9-9c29-4205-b2fb-88167bd809ec\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" Nov 03 14:29:17.499238 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:17.499196 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/941df1f9-9c29-4205-b2fb-88167bd809ec-istio-podinfo\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2\" (UID: \"941df1f9-9c29-4205-b2fb-88167bd809ec\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" Nov 03 14:29:17.499385 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:17.499249 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/941df1f9-9c29-4205-b2fb-88167bd809ec-istiod-ca-cert\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2\" (UID: \"941df1f9-9c29-4205-b2fb-88167bd809ec\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" Nov 03 14:29:17.499385 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:17.499277 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/941df1f9-9c29-4205-b2fb-88167bd809ec-workload-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2\" (UID: \"941df1f9-9c29-4205-b2fb-88167bd809ec\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" Nov 03 14:29:17.499385 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:17.499311 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/941df1f9-9c29-4205-b2fb-88167bd809ec-istio-data\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2\" (UID: \"941df1f9-9c29-4205-b2fb-88167bd809ec\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" Nov 03 14:29:17.499536 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:17.499403 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/941df1f9-9c29-4205-b2fb-88167bd809ec-credential-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2\" (UID: \"941df1f9-9c29-4205-b2fb-88167bd809ec\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" Nov 03 14:29:17.500072 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:17.500046 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/941df1f9-9c29-4205-b2fb-88167bd809ec-istiod-ca-cert\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2\" (UID: \"941df1f9-9c29-4205-b2fb-88167bd809ec\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" Nov 03 14:29:17.500188 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:17.500047 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/941df1f9-9c29-4205-b2fb-88167bd809ec-workload-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2\" (UID: \"941df1f9-9c29-4205-b2fb-88167bd809ec\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" Nov 03 14:29:17.500188 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:17.500092 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/941df1f9-9c29-4205-b2fb-88167bd809ec-workload-certs\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2\" (UID: \"941df1f9-9c29-4205-b2fb-88167bd809ec\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" Nov 03 14:29:17.501484 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:17.501444 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/941df1f9-9c29-4205-b2fb-88167bd809ec-istio-envoy\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2\" (UID: \"941df1f9-9c29-4205-b2fb-88167bd809ec\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" Nov 03 14:29:17.501939 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:17.501917 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/941df1f9-9c29-4205-b2fb-88167bd809ec-istio-podinfo\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2\" (UID: \"941df1f9-9c29-4205-b2fb-88167bd809ec\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" Nov 03 14:29:17.522432 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:17.522391 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzbj9\" (UniqueName: \"kubernetes.io/projected/941df1f9-9c29-4205-b2fb-88167bd809ec-kube-api-access-nzbj9\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2\" (UID: \"941df1f9-9c29-4205-b2fb-88167bd809ec\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" Nov 03 14:29:17.523034 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:17.522975 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/941df1f9-9c29-4205-b2fb-88167bd809ec-istio-token\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2\" (UID: \"941df1f9-9c29-4205-b2fb-88167bd809ec\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" Nov 03 14:29:17.591373 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:17.591339 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" Nov 03 14:29:17.914648 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:17.914618 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2"] Nov 03 14:29:17.916539 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:29:17.916508 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod941df1f9_9c29_4205_b2fb_88167bd809ec.slice/crio-e30c2177fbbbaf259ea24af6e4f7166a2702985d8d072a1f18ebabc9c2b0cc3d WatchSource:0}: Error finding container e30c2177fbbbaf259ea24af6e4f7166a2702985d8d072a1f18ebabc9c2b0cc3d: Status 404 returned error can't find the container with id e30c2177fbbbaf259ea24af6e4f7166a2702985d8d072a1f18ebabc9c2b0cc3d Nov 03 14:29:17.919342 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:17.919308 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:29:17.919425 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:17.919393 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:29:17.919499 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:17.919438 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:29:18.947755 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:18.947710 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" event={"ID":"941df1f9-9c29-4205-b2fb-88167bd809ec","Type":"ContainerStarted","Data":"7fda49450467c3bae83dc94c20f58447d0c2bc9916da8a2ce29e541ed4c2637b"} Nov 03 14:29:18.948233 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:18.947806 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" event={"ID":"941df1f9-9c29-4205-b2fb-88167bd809ec","Type":"ContainerStarted","Data":"e30c2177fbbbaf259ea24af6e4f7166a2702985d8d072a1f18ebabc9c2b0cc3d"} Nov 03 14:29:19.030983 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:19.030928 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" podStartSLOduration=2.030913757 podStartE2EDuration="2.030913757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:29:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:29:19.026225726 +0000 UTC m=+1092.915015320" watchObservedRunningTime="2026-11-03 14:29:19.030913757 +0000 UTC m=+1092.919703339" Nov 03 14:29:19.592205 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:19.592170 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" Nov 03 14:29:19.593757 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:19.593724 5060 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 03 14:29:19.593911 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:19.593783 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" podUID="941df1f9-9c29-4205-b2fb-88167bd809ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 03 14:29:20.592250 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:20.592215 5060 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 03 14:29:20.592652 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:20.592278 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" podUID="941df1f9-9c29-4205-b2fb-88167bd809ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 03 14:29:21.592333 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:21.592299 5060 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 03 14:29:21.592724 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:21.592363 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" podUID="941df1f9-9c29-4205-b2fb-88167bd809ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 03 14:29:22.592142 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:22.592099 5060 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 03 14:29:22.592360 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:22.592180 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" podUID="941df1f9-9c29-4205-b2fb-88167bd809ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 03 14:29:23.592429 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:23.592390 5060 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 03 14:29:23.592794 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:23.592461 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" podUID="941df1f9-9c29-4205-b2fb-88167bd809ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 03 14:29:24.767367 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:24.767308 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" Nov 03 14:29:24.767998 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:24.767742 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" Nov 03 14:29:24.769126 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:24.769080 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" Nov 03 14:29:26.045210 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:26.045175 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54c23ed0-5b8k2"] Nov 03 14:29:26.056125 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:26.056096 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54c23ed0-5b8k2"] Nov 03 14:29:26.251077 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:26.251042 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de0fc154-3d30-47e8-b036-f65b4601ae9c" path="/var/lib/kubelet/pods/de0fc154-3d30-47e8-b036-f65b4601ae9c/volumes" Nov 03 14:29:32.094342 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:32.094306 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce62b71d-mwjk2"] Nov 03 14:29:32.099996 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:32.099972 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce62b71d-mwjk2" Nov 03 14:29:32.133641 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:32.133610 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce62b71d-mwjk2"] Nov 03 14:29:32.134980 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:32.134949 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:29:32.135124 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:32.135049 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:29:32.135124 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:32.135074 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\"" Nov 03 14:29:32.135246 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:32.135144 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:29:32.228255 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:32.228217 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1b4c668-de56-4253-9f6b-bce9b7a41cfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce62b71d-mwjk2\" (UID: \"b1b4c668-de56-4253-9f6b-bce9b7a41cfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce62b71d-mwjk2" Nov 03 14:29:32.228255 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:32.228253 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb5st\" (UniqueName: \"kubernetes.io/projected/b1b4c668-de56-4253-9f6b-bce9b7a41cfc-kube-api-access-gb5st\") pod \"maas-api-revoke-sub-ce62b71d-mwjk2\" (UID: \"b1b4c668-de56-4253-9f6b-bce9b7a41cfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce62b71d-mwjk2" Nov 03 14:29:32.228571 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:32.228524 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1b4c668-de56-4253-9f6b-bce9b7a41cfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce62b71d-mwjk2\" (UID: \"b1b4c668-de56-4253-9f6b-bce9b7a41cfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce62b71d-mwjk2" Nov 03 14:29:32.330031 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:32.329979 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1b4c668-de56-4253-9f6b-bce9b7a41cfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce62b71d-mwjk2\" (UID: \"b1b4c668-de56-4253-9f6b-bce9b7a41cfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce62b71d-mwjk2" Nov 03 14:29:32.330242 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:32.330061 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1b4c668-de56-4253-9f6b-bce9b7a41cfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce62b71d-mwjk2\" (UID: \"b1b4c668-de56-4253-9f6b-bce9b7a41cfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce62b71d-mwjk2" Nov 03 14:29:32.330242 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:32.330094 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb5st\" (UniqueName: \"kubernetes.io/projected/b1b4c668-de56-4253-9f6b-bce9b7a41cfc-kube-api-access-gb5st\") pod \"maas-api-revoke-sub-ce62b71d-mwjk2\" (UID: \"b1b4c668-de56-4253-9f6b-bce9b7a41cfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce62b71d-mwjk2" Nov 03 14:29:32.330996 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:32.330966 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1b4c668-de56-4253-9f6b-bce9b7a41cfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce62b71d-mwjk2\" (UID: \"b1b4c668-de56-4253-9f6b-bce9b7a41cfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce62b71d-mwjk2" Nov 03 14:29:32.333084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:32.333056 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1b4c668-de56-4253-9f6b-bce9b7a41cfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce62b71d-mwjk2\" (UID: \"b1b4c668-de56-4253-9f6b-bce9b7a41cfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce62b71d-mwjk2" Nov 03 14:29:32.352246 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:32.352177 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb5st\" (UniqueName: \"kubernetes.io/projected/b1b4c668-de56-4253-9f6b-bce9b7a41cfc-kube-api-access-gb5st\") pod \"maas-api-revoke-sub-ce62b71d-mwjk2\" (UID: \"b1b4c668-de56-4253-9f6b-bce9b7a41cfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce62b71d-mwjk2" Nov 03 14:29:32.414944 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:32.414900 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce62b71d-mwjk2" Nov 03 14:29:32.728482 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:32.728448 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce62b71d-mwjk2"] Nov 03 14:29:32.729147 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:29:32.729118 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1b4c668_de56_4253_9f6b_bce9b7a41cfc.slice/crio-255b1bb2dabd7bb06116e0b29412e36fdda657ea66aa0a83cc8728a6a9dc1b1d WatchSource:0}: Error finding container 255b1bb2dabd7bb06116e0b29412e36fdda657ea66aa0a83cc8728a6a9dc1b1d: Status 404 returned error can't find the container with id 255b1bb2dabd7bb06116e0b29412e36fdda657ea66aa0a83cc8728a6a9dc1b1d Nov 03 14:29:33.183707 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:33.183663 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce62b71d-mwjk2" event={"ID":"b1b4c668-de56-4253-9f6b-bce9b7a41cfc","Type":"ContainerStarted","Data":"aa36f7a6421720e925dfbdc3c96f13a3f29dea58ce5ff1f53762cea4e80d2285"} Nov 03 14:29:33.184114 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:33.183728 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce62b71d-mwjk2" event={"ID":"b1b4c668-de56-4253-9f6b-bce9b7a41cfc","Type":"ContainerStarted","Data":"255b1bb2dabd7bb06116e0b29412e36fdda657ea66aa0a83cc8728a6a9dc1b1d"} Nov 03 14:29:34.192110 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:34.192079 5060 generic.go:356] "Generic (PLEG): container finished" podID="b1b4c668-de56-4253-9f6b-bce9b7a41cfc" containerID="aa36f7a6421720e925dfbdc3c96f13a3f29dea58ce5ff1f53762cea4e80d2285" exitCode=0 Nov 03 14:29:34.192517 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:34.192162 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce62b71d-mwjk2" event={"ID":"b1b4c668-de56-4253-9f6b-bce9b7a41cfc","Type":"ContainerDied","Data":"aa36f7a6421720e925dfbdc3c96f13a3f29dea58ce5ff1f53762cea4e80d2285"} Nov 03 14:29:35.358698 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:35.358664 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce62b71d-mwjk2" Nov 03 14:29:35.467384 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:35.467310 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1b4c668-de56-4253-9f6b-bce9b7a41cfc-kube-api-access-gb5st\" (UniqueName: \"kubernetes.io/projected/b1b4c668-de56-4253-9f6b-bce9b7a41cfc-kube-api-access-gb5st\") pod \"b1b4c668-de56-4253-9f6b-bce9b7a41cfc\" (UID: \"b1b4c668-de56-4253-9f6b-bce9b7a41cfc\") " Nov 03 14:29:35.467523 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:35.467411 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1b4c668-de56-4253-9f6b-bce9b7a41cfc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1b4c668-de56-4253-9f6b-bce9b7a41cfc-maas-api-openshift-ca\") pod \"b1b4c668-de56-4253-9f6b-bce9b7a41cfc\" (UID: \"b1b4c668-de56-4253-9f6b-bce9b7a41cfc\") " Nov 03 14:29:35.467523 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:35.467439 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1b4c668-de56-4253-9f6b-bce9b7a41cfc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1b4c668-de56-4253-9f6b-bce9b7a41cfc-maas-api-xks-ca\") pod \"b1b4c668-de56-4253-9f6b-bce9b7a41cfc\" (UID: \"b1b4c668-de56-4253-9f6b-bce9b7a41cfc\") " Nov 03 14:29:35.467829 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:35.467804 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1b4c668-de56-4253-9f6b-bce9b7a41cfc-maas-api-openshift-ca" pod "b1b4c668-de56-4253-9f6b-bce9b7a41cfc" (UID: "b1b4c668-de56-4253-9f6b-bce9b7a41cfc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:29:35.469461 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:35.469438 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1b4c668-de56-4253-9f6b-bce9b7a41cfc-maas-api-xks-ca" pod "b1b4c668-de56-4253-9f6b-bce9b7a41cfc" (UID: "b1b4c668-de56-4253-9f6b-bce9b7a41cfc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:29:35.469524 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:35.469446 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1b4c668-de56-4253-9f6b-bce9b7a41cfc-kube-api-access-gb5st" pod "b1b4c668-de56-4253-9f6b-bce9b7a41cfc" (UID: "b1b4c668-de56-4253-9f6b-bce9b7a41cfc"). InnerVolumeSpecName "kube-api-access-gb5st". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:29:35.569293 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:35.569256 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1b4c668-de56-4253-9f6b-bce9b7a41cfc-maas-api-openshift-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:29:35.569293 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:35.569288 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1b4c668-de56-4253-9f6b-bce9b7a41cfc-maas-api-xks-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:29:35.569293 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:35.569298 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb5st\" (UniqueName: \"kubernetes.io/projected/b1b4c668-de56-4253-9f6b-bce9b7a41cfc-kube-api-access-gb5st\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:29:36.208515 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:36.208482 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce62b71d-mwjk2" event={"ID":"b1b4c668-de56-4253-9f6b-bce9b7a41cfc","Type":"ContainerDied","Data":"255b1bb2dabd7bb06116e0b29412e36fdda657ea66aa0a83cc8728a6a9dc1b1d"} Nov 03 14:29:36.208706 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:36.208538 5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="255b1bb2dabd7bb06116e0b29412e36fdda657ea66aa0a83cc8728a6a9dc1b1d" Nov 03 14:29:36.208706 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:36.208537 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce62b71d-mwjk2" Nov 03 14:29:39.441180 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:29:39.441138 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:29:41.472737 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:41.472701 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbd38938-t48fd"] Nov 03 14:29:41.485646 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:41.485620 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbd38938-t48fd"] Nov 03 14:29:42.249194 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:42.249159 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49301a5f-e1e4-4167-8bbb-311834b37475" path="/var/lib/kubelet/pods/49301a5f-e1e4-4167-8bbb-311834b37475/volumes" Nov 03 14:29:43.108136 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:43.108101 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq"] Nov 03 14:29:43.110891 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:43.110860 5060 cpu_manager.go:414] "containerMap: removing container" podUID="b1b4c668-de56-4253-9f6b-bce9b7a41cfc" containerName="revoke-keys" Nov 03 14:29:43.111070 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:43.111050 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1b4c668-de56-4253-9f6b-bce9b7a41cfc" containerName="revoke-keys" Nov 03 14:29:43.111473 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:43.111441 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1b4c668-de56-4253-9f6b-bce9b7a41cfc" containerName="revoke-keys" Nov 03 14:29:43.117256 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:43.117233 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" Nov 03 14:29:43.125562 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:43.125533 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq"] Nov 03 14:29:43.139014 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:43.138982 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-d608daf1-gw-openshift-default-dockercfg-wmlhp\"" Nov 03 14:29:43.180037 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:43.179973 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85bc4da4-ce30-4ea6-8a43-46f361b1f611-workload-certs\") pod \"e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq\" (UID: \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\") " pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" Nov 03 14:29:43.180606 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:43.180579 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85bc4da4-ce30-4ea6-8a43-46f361b1f611-istio-envoy\") pod \"e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq\" (UID: \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\") " pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" Nov 03 14:29:43.180801 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:43.180783 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85bc4da4-ce30-4ea6-8a43-46f361b1f611-istio-data\") pod \"e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq\" (UID: \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\") " pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" Nov 03 14:29:43.182265 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:43.182245 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bc4da4-ce30-4ea6-8a43-46f361b1f611-workload-socket\") pod \"e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq\" (UID: \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\") " pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" Nov 03 14:29:43.182659 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:43.182638 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85bc4da4-ce30-4ea6-8a43-46f361b1f611-istiod-ca-cert\") pod \"e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq\" (UID: \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\") " pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" Nov 03 14:29:43.184392 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:43.184367 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85bc4da4-ce30-4ea6-8a43-46f361b1f611-istio-podinfo\") pod \"e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq\" (UID: \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\") " pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" Nov 03 14:29:43.187875 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:43.184549 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85bc4da4-ce30-4ea6-8a43-46f361b1f611-istio-token\") pod \"e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq\" (UID: \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\") " pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" Nov 03 14:29:43.187986 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:43.187918 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bc4da4-ce30-4ea6-8a43-46f361b1f611-credential-socket\") pod \"e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq\" (UID: \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\") " pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" Nov 03 14:29:43.187986 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:43.187947 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7m5d\" (UniqueName: \"kubernetes.io/projected/85bc4da4-ce30-4ea6-8a43-46f361b1f611-kube-api-access-z7m5d\") pod \"e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq\" (UID: \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\") " pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" Nov 03 14:29:43.195991 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:43.195968 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-1ffaa0/rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz"] Nov 03 14:29:43.336854 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:43.330568 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85bc4da4-ce30-4ea6-8a43-46f361b1f611-istio-token\") pod \"e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq\" (UID: \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\") " pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" Nov 03 14:29:43.336854 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:43.330624 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bc4da4-ce30-4ea6-8a43-46f361b1f611-credential-socket\") pod \"e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq\" (UID: \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\") " pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" Nov 03 14:29:43.336854 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:43.330650 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7m5d\" (UniqueName: \"kubernetes.io/projected/85bc4da4-ce30-4ea6-8a43-46f361b1f611-kube-api-access-z7m5d\") pod \"e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq\" (UID: \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\") " pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" Nov 03 14:29:43.336854 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:43.330711 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85bc4da4-ce30-4ea6-8a43-46f361b1f611-workload-certs\") pod \"e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq\" (UID: \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\") " pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" Nov 03 14:29:43.336854 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:43.330876 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85bc4da4-ce30-4ea6-8a43-46f361b1f611-istio-envoy\") pod \"e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq\" (UID: \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\") " pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" Nov 03 14:29:43.336854 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:43.330900 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85bc4da4-ce30-4ea6-8a43-46f361b1f611-istio-data\") pod \"e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq\" (UID: \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\") " pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" Nov 03 14:29:43.336854 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:43.330967 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bc4da4-ce30-4ea6-8a43-46f361b1f611-workload-socket\") pod \"e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq\" (UID: \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\") " pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" Nov 03 14:29:43.336854 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:43.331025 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85bc4da4-ce30-4ea6-8a43-46f361b1f611-istiod-ca-cert\") pod \"e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq\" (UID: \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\") " pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" Nov 03 14:29:43.336854 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:43.331058 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85bc4da4-ce30-4ea6-8a43-46f361b1f611-istio-podinfo\") pod \"e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq\" (UID: \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\") " pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" Nov 03 14:29:43.336854 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:43.332561 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bc4da4-ce30-4ea6-8a43-46f361b1f611-credential-socket\") pod \"e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq\" (UID: \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\") " pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" Nov 03 14:29:43.336854 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:43.333121 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85bc4da4-ce30-4ea6-8a43-46f361b1f611-workload-certs\") pod \"e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq\" (UID: \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\") " pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" Nov 03 14:29:43.336854 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:43.333560 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bc4da4-ce30-4ea6-8a43-46f361b1f611-workload-socket\") pod \"e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq\" (UID: \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\") " pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" Nov 03 14:29:43.346857 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:43.342344 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85bc4da4-ce30-4ea6-8a43-46f361b1f611-istio-podinfo\") pod \"e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq\" (UID: \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\") " pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" Nov 03 14:29:43.346857 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:43.344203 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85bc4da4-ce30-4ea6-8a43-46f361b1f611-istio-data\") pod \"e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq\" (UID: \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\") " pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" Nov 03 14:29:43.347517 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:43.347470 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85bc4da4-ce30-4ea6-8a43-46f361b1f611-istiod-ca-cert\") pod \"e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq\" (UID: \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\") " pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" Nov 03 14:29:43.378354 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:43.378272 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85bc4da4-ce30-4ea6-8a43-46f361b1f611-istio-envoy\") pod \"e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq\" (UID: \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\") " pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" Nov 03 14:29:43.380637 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:43.380574 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7m5d\" (UniqueName: \"kubernetes.io/projected/85bc4da4-ce30-4ea6-8a43-46f361b1f611-kube-api-access-z7m5d\") pod \"e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq\" (UID: \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\") " pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" Nov 03 14:29:43.381300 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:43.381250 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85bc4da4-ce30-4ea6-8a43-46f361b1f611-istio-token\") pod \"e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq\" (UID: \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\") " pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" Nov 03 14:29:43.442394 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:43.442071 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" Nov 03 14:29:43.889628 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:43.889590 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq"] Nov 03 14:29:43.893346 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:29:43.893314 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85bc4da4_ce30_4ea6_8a43_46f361b1f611.slice/crio-b7ca6cc7d505f9f1ef7901d66281bb54d94a89f8ab1dcb7d2a3d603516b30f0f WatchSource:0}: Error finding container b7ca6cc7d505f9f1ef7901d66281bb54d94a89f8ab1dcb7d2a3d603516b30f0f: Status 404 returned error can't find the container with id b7ca6cc7d505f9f1ef7901d66281bb54d94a89f8ab1dcb7d2a3d603516b30f0f Nov 03 14:29:43.896045 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:43.896011 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:29:43.896136 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:43.896077 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:29:43.896136 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:43.896112 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:29:44.279576 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:44.278984 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" event={"ID":"85bc4da4-ce30-4ea6-8a43-46f361b1f611","Type":"ContainerStarted","Data":"389992fe154fd4120fa10dc896e2d1cc98cca8050465774f6a5884a6d878a04a"} Nov 03 14:29:44.279576 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:44.279050 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" event={"ID":"85bc4da4-ce30-4ea6-8a43-46f361b1f611","Type":"ContainerStarted","Data":"b7ca6cc7d505f9f1ef7901d66281bb54d94a89f8ab1dcb7d2a3d603516b30f0f"} Nov 03 14:29:44.385556 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:44.385505 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" podStartSLOduration=1.385491686 podStartE2EDuration="1.385491686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:29:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:29:44.370358862 +0000 UTC m=+1118.259148445" watchObservedRunningTime="2026-11-03 14:29:44.385491686 +0000 UTC m=+1118.274281266" Nov 03 14:29:44.443199 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:44.443052 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" Nov 03 14:29:44.444508 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:44.444481 5060 patch_prober.go:29] interesting pod/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 14:29:44.444584 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:44.444553 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" podUID="85bc4da4-ce30-4ea6-8a43-46f361b1f611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 14:29:45.448523 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:45.448491 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" Nov 03 14:29:46.292361 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:46.292328 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" Nov 03 14:29:46.293243 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:46.293221 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" Nov 03 14:29:53.843861 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:53.842963 5060 patch_prober.go:29] interesting pod/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": read tcp 10.132.0.2:45306->10.132.0.92:15021: read: connection reset by peer" start-of-body= Nov 03 14:29:53.843861 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:53.843053 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" podUID="07fa990f-370e-4d84-b7e6-340f4caa6348" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": read tcp 10.132.0.2:45306->10.132.0.92:15021: read: connection reset by peer" Nov 03 14:29:54.491859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:54.491792 5060 generic.go:356] "Generic (PLEG): container finished" podID="07fa990f-370e-4d84-b7e6-340f4caa6348" containerID="f16df8daebf0a320416c241bacc909170a392e9e7159285076f80439fc860ac9" exitCode=0 Nov 03 14:29:54.492068 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:54.491952 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" event={"ID":"07fa990f-370e-4d84-b7e6-340f4caa6348","Type":"ContainerDied","Data":"f16df8daebf0a320416c241bacc909170a392e9e7159285076f80439fc860ac9"} Nov 03 14:29:54.495572 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:54.492480 5060 scope.go:122] "RemoveContainer" containerID="f16df8daebf0a320416c241bacc909170a392e9e7159285076f80439fc860ac9" Nov 03 14:29:54.496758 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:54.496718 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:29:54.496892 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:54.496813 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:29:54.496968 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:54.496902 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:29:55.500738 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:55.500703 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" event={"ID":"07fa990f-370e-4d84-b7e6-340f4caa6348","Type":"ContainerStarted","Data":"73f3789700b9b3481bf161f6f8c3a04b171b4965563257f488c66bcdbe9f5452"} Nov 03 14:29:56.330140 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:56.330094 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" Nov 03 14:29:56.330480 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:56.330355 5060 patch_prober.go:29] interesting pod/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 14:29:56.330480 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:56.330411 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" podUID="07fa990f-370e-4d84-b7e6-340f4caa6348" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 14:29:57.047603 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:57.047552 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6e1e5a0-ng7sx"] Nov 03 14:29:57.058486 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:57.058448 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6e1e5a0-ng7sx"] Nov 03 14:29:57.330084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:57.329999 5060 patch_prober.go:29] interesting pod/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 14:29:57.330084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:57.330065 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" podUID="07fa990f-370e-4d84-b7e6-340f4caa6348" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 14:29:58.235895 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:58.235846 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz" podUID="930bcc45-2ffe-4502-aff6-03761f25ee62" containerName="main" containerID="cri-o://6039b6e672816c985ab88f05f2940ccdf4de8db764d78ce2a15d546a528abd64" gracePeriod=45 Nov 03 14:29:58.250244 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:58.250214 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eeae7683-bed5-41bb-addf-27c59d5f2860" path="/var/lib/kubelet/pods/eeae7683-bed5-41bb-addf-27c59d5f2860/volumes" Nov 03 14:29:58.330619 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:58.330585 5060 patch_prober.go:29] interesting pod/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 14:29:58.330814 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:58.330646 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" podUID="07fa990f-370e-4d84-b7e6-340f4caa6348" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 14:29:58.581540 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:58.581509 5060 generic.go:356] "Generic (PLEG): container finished" podID="930bcc45-2ffe-4502-aff6-03761f25ee62" containerID="6039b6e672816c985ab88f05f2940ccdf4de8db764d78ce2a15d546a528abd64" exitCode=0 Nov 03 14:29:58.581683 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:58.581599 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz" event={"ID":"930bcc45-2ffe-4502-aff6-03761f25ee62","Type":"ContainerDied","Data":"6039b6e672816c985ab88f05f2940ccdf4de8db764d78ce2a15d546a528abd64"} Nov 03 14:29:58.648735 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:58.647428 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz" Nov 03 14:29:58.777086 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:58.777056 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/930bcc45-2ffe-4502-aff6-03761f25ee62-tls-certs\" (UniqueName: \"kubernetes.io/secret/930bcc45-2ffe-4502-aff6-03761f25ee62-tls-certs\") pod \"930bcc45-2ffe-4502-aff6-03761f25ee62\" (UID: \"930bcc45-2ffe-4502-aff6-03761f25ee62\") " Nov 03 14:29:58.777255 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:58.777102 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/930bcc45-2ffe-4502-aff6-03761f25ee62-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/930bcc45-2ffe-4502-aff6-03761f25ee62-model-cache\") pod \"930bcc45-2ffe-4502-aff6-03761f25ee62\" (UID: \"930bcc45-2ffe-4502-aff6-03761f25ee62\") " Nov 03 14:29:58.777255 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:58.777176 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/930bcc45-2ffe-4502-aff6-03761f25ee62-dshm\" (UniqueName: \"kubernetes.io/empty-dir/930bcc45-2ffe-4502-aff6-03761f25ee62-dshm\") pod \"930bcc45-2ffe-4502-aff6-03761f25ee62\" (UID: \"930bcc45-2ffe-4502-aff6-03761f25ee62\") " Nov 03 14:29:58.777255 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:58.777244 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/930bcc45-2ffe-4502-aff6-03761f25ee62-home\" (UniqueName: \"kubernetes.io/empty-dir/930bcc45-2ffe-4502-aff6-03761f25ee62-home\") pod \"930bcc45-2ffe-4502-aff6-03761f25ee62\" (UID: \"930bcc45-2ffe-4502-aff6-03761f25ee62\") " Nov 03 14:29:58.777385 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:58.777292 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/930bcc45-2ffe-4502-aff6-03761f25ee62-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/930bcc45-2ffe-4502-aff6-03761f25ee62-tmp-dir\") pod \"930bcc45-2ffe-4502-aff6-03761f25ee62\" (UID: \"930bcc45-2ffe-4502-aff6-03761f25ee62\") " Nov 03 14:29:58.777385 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:58.777362 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/930bcc45-2ffe-4502-aff6-03761f25ee62-kube-api-access-v77lx\" (UniqueName: \"kubernetes.io/projected/930bcc45-2ffe-4502-aff6-03761f25ee62-kube-api-access-v77lx\") pod \"930bcc45-2ffe-4502-aff6-03761f25ee62\" (UID: \"930bcc45-2ffe-4502-aff6-03761f25ee62\") " Nov 03 14:29:58.777459 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:58.777425 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/930bcc45-2ffe-4502-aff6-03761f25ee62-model-cache" pod "930bcc45-2ffe-4502-aff6-03761f25ee62" (UID: "930bcc45-2ffe-4502-aff6-03761f25ee62"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:29:58.777596 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:58.777552 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/930bcc45-2ffe-4502-aff6-03761f25ee62-home" pod "930bcc45-2ffe-4502-aff6-03761f25ee62" (UID: "930bcc45-2ffe-4502-aff6-03761f25ee62"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:29:58.777737 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:58.777621 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/930bcc45-2ffe-4502-aff6-03761f25ee62-tmp-dir" pod "930bcc45-2ffe-4502-aff6-03761f25ee62" (UID: "930bcc45-2ffe-4502-aff6-03761f25ee62"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:29:58.777850 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:58.777821 5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/930bcc45-2ffe-4502-aff6-03761f25ee62-model-cache\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:29:58.778018 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:58.777997 5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/930bcc45-2ffe-4502-aff6-03761f25ee62-home\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:29:58.778112 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:58.778023 5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/930bcc45-2ffe-4502-aff6-03761f25ee62-tmp-dir\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:29:58.779419 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:58.779391 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/930bcc45-2ffe-4502-aff6-03761f25ee62-dshm" pod "930bcc45-2ffe-4502-aff6-03761f25ee62" (UID: "930bcc45-2ffe-4502-aff6-03761f25ee62"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:29:58.779517 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:58.779492 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/930bcc45-2ffe-4502-aff6-03761f25ee62-tls-certs" pod "930bcc45-2ffe-4502-aff6-03761f25ee62" (UID: "930bcc45-2ffe-4502-aff6-03761f25ee62"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:29:58.779966 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:58.779944 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/930bcc45-2ffe-4502-aff6-03761f25ee62-kube-api-access-v77lx" pod "930bcc45-2ffe-4502-aff6-03761f25ee62" (UID: "930bcc45-2ffe-4502-aff6-03761f25ee62"). InnerVolumeSpecName "kube-api-access-v77lx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:29:58.878963 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:58.878879 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v77lx\" (UniqueName: \"kubernetes.io/projected/930bcc45-2ffe-4502-aff6-03761f25ee62-kube-api-access-v77lx\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:29:58.878963 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:58.878910 5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/930bcc45-2ffe-4502-aff6-03761f25ee62-tls-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:29:58.878963 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:58.878918 5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/930bcc45-2ffe-4502-aff6-03761f25ee62-dshm\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:29:59.330650 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:59.330616 5060 patch_prober.go:29] interesting pod/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 14:29:59.331132 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:59.330668 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" podUID="07fa990f-370e-4d84-b7e6-340f4caa6348" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 14:29:59.594628 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:59.594543 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz" Nov 03 14:29:59.594780 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:59.594632 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz" event={"ID":"930bcc45-2ffe-4502-aff6-03761f25ee62","Type":"ContainerDied","Data":"7c2a51a397fdba01b2f9e0aaa02cfa26c14537fc7f7e6e54833ff12f712fa94d"} Nov 03 14:29:59.594780 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:59.594678 5060 scope.go:122] "RemoveContainer" containerID="6039b6e672816c985ab88f05f2940ccdf4de8db764d78ce2a15d546a528abd64" Nov 03 14:29:59.663355 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:59.663321 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-1ffaa0/rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz"] Nov 03 14:29:59.675497 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:59.675466 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-1ffaa0/rate-test-model-1ffaa0-kserve-6cddd44c7c-k6djz"] Nov 03 14:29:59.845247 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:59.845165 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-4a67e25f/e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p"] Nov 03 14:29:59.846290 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:59.846247 5060 cpu_manager.go:414] "containerMap: removing container" podUID="930bcc45-2ffe-4502-aff6-03761f25ee62" containerName="main" Nov 03 14:29:59.846467 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:59.846453 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="930bcc45-2ffe-4502-aff6-03761f25ee62" containerName="main" Nov 03 14:29:59.846713 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:59.846701 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="930bcc45-2ffe-4502-aff6-03761f25ee62" containerName="main" Nov 03 14:29:59.866265 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:59.860315 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-4a67e25f/e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p"] Nov 03 14:29:59.866265 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:59.860429 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4a67e25f/e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p" Nov 03 14:29:59.878793 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:59.878766 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-4a67e25f\"/\"e2e-lifecycle-model-4a67e25f-kserve-self-signed-certs\"" Nov 03 14:29:59.878793 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:59.878769 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-4a67e25f\"/\"default-dockercfg-xt8bf\"" Nov 03 14:29:59.879029 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:59.878800 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-4a67e25f\"/\"openshift-service-ca.crt\"" Nov 03 14:29:59.879029 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:59.878887 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-4a67e25f\"/\"kube-root-ca.crt\"" Nov 03 14:29:59.994787 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:59.994735 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f-model-cache\") pod \"e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p\" (UID: \"0476cf91-73a3-4392-a1c7-1dc62e2a5b9f\") " pod="ai-tenant-e2e-mt-4a67e25f/e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p" Nov 03 14:29:59.995012 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:59.994887 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f-dshm\") pod \"e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p\" (UID: \"0476cf91-73a3-4392-a1c7-1dc62e2a5b9f\") " pod="ai-tenant-e2e-mt-4a67e25f/e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p" Nov 03 14:29:59.995080 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:59.995044 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f-tmp-dir\") pod \"e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p\" (UID: \"0476cf91-73a3-4392-a1c7-1dc62e2a5b9f\") " pod="ai-tenant-e2e-mt-4a67e25f/e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p" Nov 03 14:29:59.995135 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:59.995096 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f-home\") pod \"e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p\" (UID: \"0476cf91-73a3-4392-a1c7-1dc62e2a5b9f\") " pod="ai-tenant-e2e-mt-4a67e25f/e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p" Nov 03 14:29:59.995209 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:59.995193 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f-tls-certs\") pod \"e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p\" (UID: \"0476cf91-73a3-4392-a1c7-1dc62e2a5b9f\") " pod="ai-tenant-e2e-mt-4a67e25f/e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p" Nov 03 14:29:59.995272 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:29:59.995228 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-997ct\" (UniqueName: \"kubernetes.io/projected/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f-kube-api-access-997ct\") pod \"e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p\" (UID: \"0476cf91-73a3-4392-a1c7-1dc62e2a5b9f\") " pod="ai-tenant-e2e-mt-4a67e25f/e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p" Nov 03 14:30:00.097919 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.097527 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f-dshm\") pod \"e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p\" (UID: \"0476cf91-73a3-4392-a1c7-1dc62e2a5b9f\") " pod="ai-tenant-e2e-mt-4a67e25f/e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p" Nov 03 14:30:00.097919 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.097670 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f-tmp-dir\") pod \"e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p\" (UID: \"0476cf91-73a3-4392-a1c7-1dc62e2a5b9f\") " pod="ai-tenant-e2e-mt-4a67e25f/e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p" Nov 03 14:30:00.097919 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.097718 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f-home\") pod \"e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p\" (UID: \"0476cf91-73a3-4392-a1c7-1dc62e2a5b9f\") " pod="ai-tenant-e2e-mt-4a67e25f/e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p" Nov 03 14:30:00.097919 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.097758 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f-tls-certs\") pod \"e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p\" (UID: \"0476cf91-73a3-4392-a1c7-1dc62e2a5b9f\") " pod="ai-tenant-e2e-mt-4a67e25f/e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p" Nov 03 14:30:00.097919 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.097786 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-997ct\" (UniqueName: \"kubernetes.io/projected/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f-kube-api-access-997ct\") pod \"e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p\" (UID: \"0476cf91-73a3-4392-a1c7-1dc62e2a5b9f\") " pod="ai-tenant-e2e-mt-4a67e25f/e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p" Nov 03 14:30:00.100029 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.099994 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f-model-cache\") pod \"e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p\" (UID: \"0476cf91-73a3-4392-a1c7-1dc62e2a5b9f\") " pod="ai-tenant-e2e-mt-4a67e25f/e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p" Nov 03 14:30:00.100651 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.100627 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f-model-cache\") pod \"e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p\" (UID: \"0476cf91-73a3-4392-a1c7-1dc62e2a5b9f\") " pod="ai-tenant-e2e-mt-4a67e25f/e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p" Nov 03 14:30:00.101083 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.101062 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f-home\") pod \"e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p\" (UID: \"0476cf91-73a3-4392-a1c7-1dc62e2a5b9f\") " pod="ai-tenant-e2e-mt-4a67e25f/e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p" Nov 03 14:30:00.101443 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.101424 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f-tmp-dir\") pod \"e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p\" (UID: \"0476cf91-73a3-4392-a1c7-1dc62e2a5b9f\") " pod="ai-tenant-e2e-mt-4a67e25f/e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p" Nov 03 14:30:00.123048 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.123018 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f-tls-certs\") pod \"e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p\" (UID: \"0476cf91-73a3-4392-a1c7-1dc62e2a5b9f\") " pod="ai-tenant-e2e-mt-4a67e25f/e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p" Nov 03 14:30:00.134514 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.127310 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-997ct\" (UniqueName: \"kubernetes.io/projected/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f-kube-api-access-997ct\") pod \"e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p\" (UID: \"0476cf91-73a3-4392-a1c7-1dc62e2a5b9f\") " pod="ai-tenant-e2e-mt-4a67e25f/e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p" Nov 03 14:30:00.164858 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.160125 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f-dshm\") pod \"e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p\" (UID: \"0476cf91-73a3-4392-a1c7-1dc62e2a5b9f\") " pod="ai-tenant-e2e-mt-4a67e25f/e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p" Nov 03 14:30:00.190441 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.190185 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4a67e25f/e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p" Nov 03 14:30:00.343002 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.342945 5060 patch_prober.go:29] interesting pod/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 14:30:00.343511 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.343049 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" podUID="07fa990f-370e-4d84-b7e6-340f4caa6348" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 14:30:00.368697 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.368607 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="930bcc45-2ffe-4502-aff6-03761f25ee62" path="/var/lib/kubelet/pods/930bcc45-2ffe-4502-aff6-03761f25ee62/volumes" Nov 03 14:30:00.369607 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.369582 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1ffaa0-29895270-xc62v"] Nov 03 14:30:00.378012 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.377991 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895270-gg8xh"] Nov 03 14:30:00.379525 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.379505 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1ffaa0-29895270-xc62v" Nov 03 14:30:00.382923 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.382900 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1ffaa0-29895270-xc62v"] Nov 03 14:30:00.383032 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.382936 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4a67e25f-29895270-mzh9z"] Nov 03 14:30:00.383032 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.383010 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-gg8xh" Nov 03 14:30:00.386933 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.386709 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895270-grzgd"] Nov 03 14:30:00.388893 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.388652 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4a67e25f-29895270-mzh9z" Nov 03 14:30:00.394925 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.391799 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895270-gg8xh"] Nov 03 14:30:00.394925 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.391875 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895270-grzgd"] Nov 03 14:30:00.394925 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.391902 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895270-grzgd" Nov 03 14:30:00.394925 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.391906 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4a67e25f-29895270-mzh9z"] Nov 03 14:30:00.397043 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.397019 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\"" Nov 03 14:30:00.397171 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.397155 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:30:00.397247 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.397197 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:30:00.449083 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.448889 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thldh\" (UniqueName: \"kubernetes.io/projected/f0590e2b-4337-4428-bc50-a2f1d54c289c-kube-api-access-thldh\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-1ffaa0-29895270-xc62v\" (UID: \"f0590e2b-4337-4428-bc50-a2f1d54c289c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1ffaa0-29895270-xc62v" Nov 03 14:30:00.449083 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.449054 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a39db37e-e5c9-4b6c-b989-cf0fc9cedacc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-4a67e25f-29895270-mzh9z\" (UID: \"a39db37e-e5c9-4b6c-b989-cf0fc9cedacc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4a67e25f-29895270-mzh9z" Nov 03 14:30:00.453879 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.449333 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddcjg\" (UniqueName: \"kubernetes.io/projected/59925bb5-ef87-4382-a542-9134ddb53466-kube-api-access-ddcjg\") pod \"maas-api-key-cleanup-e2e-extmod-b-b4361f-29895270-grzgd\" (UID: \"59925bb5-ef87-4382-a542-9134ddb53466\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895270-grzgd" Nov 03 14:30:00.453879 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.449824 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7zww\" (UniqueName: \"kubernetes.io/projected/a39db37e-e5c9-4b6c-b989-cf0fc9cedacc-kube-api-access-v7zww\") pod \"maas-api-key-cleanup-e2e-mt-4a67e25f-29895270-mzh9z\" (UID: \"a39db37e-e5c9-4b6c-b989-cf0fc9cedacc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4a67e25f-29895270-mzh9z" Nov 03 14:30:00.453879 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.449966 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73ee53fb-ff99-4bee-b2fa-c5e2c5874a68-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895270-gg8xh\" (UID: \"73ee53fb-ff99-4bee-b2fa-c5e2c5874a68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-gg8xh" Nov 03 14:30:00.453879 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.450177 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59925bb5-ef87-4382-a542-9134ddb53466-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-b4361f-29895270-grzgd\" (UID: \"59925bb5-ef87-4382-a542-9134ddb53466\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895270-grzgd" Nov 03 14:30:00.453879 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.450349 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0590e2b-4337-4428-bc50-a2f1d54c289c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-1ffaa0-29895270-xc62v\" (UID: \"f0590e2b-4337-4428-bc50-a2f1d54c289c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1ffaa0-29895270-xc62v" Nov 03 14:30:00.453879 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.450530 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lddlc\" (UniqueName: \"kubernetes.io/projected/73ee53fb-ff99-4bee-b2fa-c5e2c5874a68-kube-api-access-lddlc\") pod \"maas-api-key-cleanup-29895270-gg8xh\" (UID: \"73ee53fb-ff99-4bee-b2fa-c5e2c5874a68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-gg8xh" Nov 03 14:30:00.552025 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.551988 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0590e2b-4337-4428-bc50-a2f1d54c289c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-1ffaa0-29895270-xc62v\" (UID: \"f0590e2b-4337-4428-bc50-a2f1d54c289c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1ffaa0-29895270-xc62v" Nov 03 14:30:00.552025 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.552025 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lddlc\" (UniqueName: \"kubernetes.io/projected/73ee53fb-ff99-4bee-b2fa-c5e2c5874a68-kube-api-access-lddlc\") pod \"maas-api-key-cleanup-29895270-gg8xh\" (UID: \"73ee53fb-ff99-4bee-b2fa-c5e2c5874a68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-gg8xh" Nov 03 14:30:00.552295 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.552080 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thldh\" (UniqueName: \"kubernetes.io/projected/f0590e2b-4337-4428-bc50-a2f1d54c289c-kube-api-access-thldh\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-1ffaa0-29895270-xc62v\" (UID: \"f0590e2b-4337-4428-bc50-a2f1d54c289c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1ffaa0-29895270-xc62v" Nov 03 14:30:00.552295 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.552130 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a39db37e-e5c9-4b6c-b989-cf0fc9cedacc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-4a67e25f-29895270-mzh9z\" (UID: \"a39db37e-e5c9-4b6c-b989-cf0fc9cedacc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4a67e25f-29895270-mzh9z" Nov 03 14:30:00.552295 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.552162 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddcjg\" (UniqueName: \"kubernetes.io/projected/59925bb5-ef87-4382-a542-9134ddb53466-kube-api-access-ddcjg\") pod \"maas-api-key-cleanup-e2e-extmod-b-b4361f-29895270-grzgd\" (UID: \"59925bb5-ef87-4382-a542-9134ddb53466\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895270-grzgd" Nov 03 14:30:00.552295 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.552287 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7zww\" (UniqueName: \"kubernetes.io/projected/a39db37e-e5c9-4b6c-b989-cf0fc9cedacc-kube-api-access-v7zww\") pod \"maas-api-key-cleanup-e2e-mt-4a67e25f-29895270-mzh9z\" (UID: \"a39db37e-e5c9-4b6c-b989-cf0fc9cedacc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4a67e25f-29895270-mzh9z" Nov 03 14:30:00.552499 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.552317 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73ee53fb-ff99-4bee-b2fa-c5e2c5874a68-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895270-gg8xh\" (UID: \"73ee53fb-ff99-4bee-b2fa-c5e2c5874a68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-gg8xh" Nov 03 14:30:00.552499 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.552381 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59925bb5-ef87-4382-a542-9134ddb53466-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-b4361f-29895270-grzgd\" (UID: \"59925bb5-ef87-4382-a542-9134ddb53466\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895270-grzgd" Nov 03 14:30:00.552867 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.552824 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0590e2b-4337-4428-bc50-a2f1d54c289c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-1ffaa0-29895270-xc62v\" (UID: \"f0590e2b-4337-4428-bc50-a2f1d54c289c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1ffaa0-29895270-xc62v" Nov 03 14:30:00.552935 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.552830 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a39db37e-e5c9-4b6c-b989-cf0fc9cedacc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-4a67e25f-29895270-mzh9z\" (UID: \"a39db37e-e5c9-4b6c-b989-cf0fc9cedacc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4a67e25f-29895270-mzh9z" Nov 03 14:30:00.553092 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.553074 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59925bb5-ef87-4382-a542-9134ddb53466-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-b4361f-29895270-grzgd\" (UID: \"59925bb5-ef87-4382-a542-9134ddb53466\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895270-grzgd" Nov 03 14:30:00.553255 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.553231 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73ee53fb-ff99-4bee-b2fa-c5e2c5874a68-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895270-gg8xh\" (UID: \"73ee53fb-ff99-4bee-b2fa-c5e2c5874a68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-gg8xh" Nov 03 14:30:00.576902 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.576866 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thldh\" (UniqueName: \"kubernetes.io/projected/f0590e2b-4337-4428-bc50-a2f1d54c289c-kube-api-access-thldh\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-1ffaa0-29895270-xc62v\" (UID: \"f0590e2b-4337-4428-bc50-a2f1d54c289c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1ffaa0-29895270-xc62v" Nov 03 14:30:00.577461 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.577440 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7zww\" (UniqueName: \"kubernetes.io/projected/a39db37e-e5c9-4b6c-b989-cf0fc9cedacc-kube-api-access-v7zww\") pod \"maas-api-key-cleanup-e2e-mt-4a67e25f-29895270-mzh9z\" (UID: \"a39db37e-e5c9-4b6c-b989-cf0fc9cedacc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4a67e25f-29895270-mzh9z" Nov 03 14:30:00.577629 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.577475 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lddlc\" (UniqueName: \"kubernetes.io/projected/73ee53fb-ff99-4bee-b2fa-c5e2c5874a68-kube-api-access-lddlc\") pod \"maas-api-key-cleanup-29895270-gg8xh\" (UID: \"73ee53fb-ff99-4bee-b2fa-c5e2c5874a68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-gg8xh" Nov 03 14:30:00.578039 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.578019 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddcjg\" (UniqueName: \"kubernetes.io/projected/59925bb5-ef87-4382-a542-9134ddb53466-kube-api-access-ddcjg\") pod \"maas-api-key-cleanup-e2e-extmod-b-b4361f-29895270-grzgd\" (UID: \"59925bb5-ef87-4382-a542-9134ddb53466\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895270-grzgd" Nov 03 14:30:00.659461 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.659430 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-4a67e25f/e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p"] Nov 03 14:30:00.659883 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:30:00.659854 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0476cf91_73a3_4392_a1c7_1dc62e2a5b9f.slice/crio-bb3668af23722e77664db87101b4567deca610001e8563fad5894fa09f57e145 WatchSource:0}: Error finding container bb3668af23722e77664db87101b4567deca610001e8563fad5894fa09f57e145: Status 404 returned error can't find the container with id bb3668af23722e77664db87101b4567deca610001e8563fad5894fa09f57e145 Nov 03 14:30:00.736100 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.736072 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1ffaa0-29895270-xc62v" Nov 03 14:30:00.760978 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.760952 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-gg8xh" Nov 03 14:30:00.771227 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.770240 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4a67e25f-29895270-mzh9z" Nov 03 14:30:00.775493 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:00.775471 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895270-grzgd" Nov 03 14:30:01.078426 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:01.078028 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1ffaa0-29895270-xc62v"] Nov 03 14:30:01.078825 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:30:01.078644 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0590e2b_4337_4428_bc50_a2f1d54c289c.slice/crio-db166d48aeb25ba8245bec66ccdf0b2e1a0d21704706b6794b2a81c0dd788450 WatchSource:0}: Error finding container db166d48aeb25ba8245bec66ccdf0b2e1a0d21704706b6794b2a81c0dd788450: Status 404 returned error can't find the container with id db166d48aeb25ba8245bec66ccdf0b2e1a0d21704706b6794b2a81c0dd788450 Nov 03 14:30:01.133060 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:01.127395 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895270-gg8xh"] Nov 03 14:30:01.135788 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:30:01.135729 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73ee53fb_ff99_4bee_b2fa_c5e2c5874a68.slice/crio-85ac8443a58392d03b685e58280c2ea170514d3b113f0ecb3a6998005d6e5277 WatchSource:0}: Error finding container 85ac8443a58392d03b685e58280c2ea170514d3b113f0ecb3a6998005d6e5277: Status 404 returned error can't find the container with id 85ac8443a58392d03b685e58280c2ea170514d3b113f0ecb3a6998005d6e5277 Nov 03 14:30:01.216354 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:01.216327 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895270-grzgd"] Nov 03 14:30:01.224361 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:01.224333 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4a67e25f-29895270-mzh9z"] Nov 03 14:30:01.427566 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:01.427526 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" Nov 03 14:30:01.428938 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:01.428878 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" Nov 03 14:30:01.434158 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:01.434086 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" Nov 03 14:30:01.622713 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:01.622673 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-gg8xh" event={"ID":"73ee53fb-ff99-4bee-b2fa-c5e2c5874a68","Type":"ContainerStarted","Data":"0aade806bc7f74d193c2a3f613a7b7a9c59e7d4e8d5eb83e4b67cd86ae84820d"} Nov 03 14:30:01.622927 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:01.622745 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-gg8xh" event={"ID":"73ee53fb-ff99-4bee-b2fa-c5e2c5874a68","Type":"ContainerStarted","Data":"85ac8443a58392d03b685e58280c2ea170514d3b113f0ecb3a6998005d6e5277"} Nov 03 14:30:01.624500 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:01.624467 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4a67e25f/e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p" event={"ID":"0476cf91-73a3-4392-a1c7-1dc62e2a5b9f","Type":"ContainerStarted","Data":"1f57cbde036137476a25b7f3998c5023bd9ba2846cf317aaf124e46a4362a312"} Nov 03 14:30:01.624500 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:01.624502 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4a67e25f/e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p" event={"ID":"0476cf91-73a3-4392-a1c7-1dc62e2a5b9f","Type":"ContainerStarted","Data":"bb3668af23722e77664db87101b4567deca610001e8563fad5894fa09f57e145"} Nov 03 14:30:01.625749 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:01.625728 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895270-grzgd" event={"ID":"59925bb5-ef87-4382-a542-9134ddb53466","Type":"ContainerStarted","Data":"bd11a5678617033b7f08e1fcb49ffd59fac999d6de99dfc7132fa69655b4591d"} Nov 03 14:30:01.626981 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:01.626957 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4a67e25f-29895270-mzh9z" event={"ID":"a39db37e-e5c9-4b6c-b989-cf0fc9cedacc","Type":"ContainerStarted","Data":"b17087ef59ed3a7fa8d706ec3347ca42062e853b148632c44da5485739f72c6a"} Nov 03 14:30:01.628526 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:01.628508 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1ffaa0-29895270-xc62v" event={"ID":"f0590e2b-4337-4428-bc50-a2f1d54c289c","Type":"ContainerStarted","Data":"c9a2d89ae3ca843a3e48215e236cc0a0c24a7defd2de6717f3abaaa59f85bf00"} Nov 03 14:30:01.628610 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:01.628538 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1ffaa0-29895270-xc62v" event={"ID":"f0590e2b-4337-4428-bc50-a2f1d54c289c","Type":"ContainerStarted","Data":"db166d48aeb25ba8245bec66ccdf0b2e1a0d21704706b6794b2a81c0dd788450"} Nov 03 14:30:01.669794 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:01.669731 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-gg8xh" podStartSLOduration=1.6697145660000001 podStartE2EDuration="1.669714566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:30:01.665436022 +0000 UTC m=+1135.554225604" watchObservedRunningTime="2026-11-03 14:30:01.669714566 +0000 UTC m=+1135.558504148" Nov 03 14:30:01.711650 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:01.711606 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-4a67e25f/e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p" podStartSLOduration=2.711591352 podStartE2EDuration="2.711591352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:29:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:30:01.708776465 +0000 UTC m=+1135.597566046" watchObservedRunningTime="2026-11-03 14:30:01.711591352 +0000 UTC m=+1135.600380933" Nov 03 14:30:01.756871 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:01.756804 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1ffaa0-29895270-xc62v" podStartSLOduration=1.756791079 podStartE2EDuration="1.756791079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:30:01.753231678 +0000 UTC m=+1135.642021270" watchObservedRunningTime="2026-11-03 14:30:01.756791079 +0000 UTC m=+1135.645580660" Nov 03 14:30:02.190991 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:02.190959 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-4a67e25f/e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p" Nov 03 14:30:02.642144 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:02.642057 5060 generic.go:356] "Generic (PLEG): container finished" podID="f0590e2b-4337-4428-bc50-a2f1d54c289c" containerID="c9a2d89ae3ca843a3e48215e236cc0a0c24a7defd2de6717f3abaaa59f85bf00" exitCode=60 Nov 03 14:30:02.642587 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:02.642154 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1ffaa0-29895270-xc62v" event={"ID":"f0590e2b-4337-4428-bc50-a2f1d54c289c","Type":"ContainerDied","Data":"c9a2d89ae3ca843a3e48215e236cc0a0c24a7defd2de6717f3abaaa59f85bf00"} Nov 03 14:30:02.642587 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:02.642458 5060 scope.go:122] "RemoveContainer" containerID="c9a2d89ae3ca843a3e48215e236cc0a0c24a7defd2de6717f3abaaa59f85bf00" Nov 03 14:30:02.644165 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:02.644126 5060 generic.go:356] "Generic (PLEG): container finished" podID="73ee53fb-ff99-4bee-b2fa-c5e2c5874a68" containerID="0aade806bc7f74d193c2a3f613a7b7a9c59e7d4e8d5eb83e4b67cd86ae84820d" exitCode=60 Nov 03 14:30:02.644230 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:02.644204 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-gg8xh" event={"ID":"73ee53fb-ff99-4bee-b2fa-c5e2c5874a68","Type":"ContainerDied","Data":"0aade806bc7f74d193c2a3f613a7b7a9c59e7d4e8d5eb83e4b67cd86ae84820d"} Nov 03 14:30:02.644572 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:02.644549 5060 scope.go:122] "RemoveContainer" containerID="0aade806bc7f74d193c2a3f613a7b7a9c59e7d4e8d5eb83e4b67cd86ae84820d" Nov 03 14:30:02.646331 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:02.646266 5060 generic.go:356] "Generic (PLEG): container finished" podID="59925bb5-ef87-4382-a542-9134ddb53466" containerID="c1d7699880965a579036b9acaa2d1a4ce35ada6effb96b1fbfa25c19e0444a67" exitCode=60 Nov 03 14:30:02.646456 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:02.646325 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895270-grzgd" event={"ID":"59925bb5-ef87-4382-a542-9134ddb53466","Type":"ContainerDied","Data":"c1d7699880965a579036b9acaa2d1a4ce35ada6effb96b1fbfa25c19e0444a67"} Nov 03 14:30:02.646592 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:02.646575 5060 scope.go:122] "RemoveContainer" containerID="c1d7699880965a579036b9acaa2d1a4ce35ada6effb96b1fbfa25c19e0444a67" Nov 03 14:30:02.648083 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:02.648051 5060 generic.go:356] "Generic (PLEG): container finished" podID="a39db37e-e5c9-4b6c-b989-cf0fc9cedacc" containerID="e52e0b704cb4bfa70c759bb4887ebdefa4a19f9b8953352422ff51d23c1c898e" exitCode=60 Nov 03 14:30:02.648142 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:02.648097 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4a67e25f-29895270-mzh9z" event={"ID":"a39db37e-e5c9-4b6c-b989-cf0fc9cedacc","Type":"ContainerDied","Data":"e52e0b704cb4bfa70c759bb4887ebdefa4a19f9b8953352422ff51d23c1c898e"} Nov 03 14:30:02.648303 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:02.648289 5060 scope.go:122] "RemoveContainer" containerID="e52e0b704cb4bfa70c759bb4887ebdefa4a19f9b8953352422ff51d23c1c898e" Nov 03 14:30:03.043621 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:03.043583 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8116d673-lnczp"] Nov 03 14:30:03.047705 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:03.047654 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8116d673-lnczp"] Nov 03 14:30:03.657586 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:03.657548 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-gg8xh" event={"ID":"73ee53fb-ff99-4bee-b2fa-c5e2c5874a68","Type":"ContainerStarted","Data":"83b0636c0019ba6eecb69ceeb0f9faabd6de52ce3fc76bc3d9a1a2cfda2baa1a"} Nov 03 14:30:03.659594 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:03.659561 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895270-grzgd" event={"ID":"59925bb5-ef87-4382-a542-9134ddb53466","Type":"ContainerStarted","Data":"fbf59c9c41778fecd38eca57fb9764de32edb3305bfbda389d93c4dbd777a634"} Nov 03 14:30:03.661556 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:03.661534 5060 generic.go:356] "Generic (PLEG): container finished" podID="a39db37e-e5c9-4b6c-b989-cf0fc9cedacc" containerID="bbf0408a7e33589d3567ce5ee6ce2721b91826da2410b8f2a0dec590fb21e5e5" exitCode=60 Nov 03 14:30:03.661649 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:03.661584 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4a67e25f-29895270-mzh9z" event={"ID":"a39db37e-e5c9-4b6c-b989-cf0fc9cedacc","Type":"ContainerDied","Data":"bbf0408a7e33589d3567ce5ee6ce2721b91826da2410b8f2a0dec590fb21e5e5"} Nov 03 14:30:03.661649 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:03.661639 5060 scope.go:122] "RemoveContainer" containerID="e52e0b704cb4bfa70c759bb4887ebdefa4a19f9b8953352422ff51d23c1c898e" Nov 03 14:30:03.661922 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:03.661907 5060 scope.go:122] "RemoveContainer" containerID="bbf0408a7e33589d3567ce5ee6ce2721b91826da2410b8f2a0dec590fb21e5e5" Nov 03 14:30:03.662203 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:30:03.662185 5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-4a67e25f-29895270-mzh9z_odh-ai-gateway-infra(a39db37e-e5c9-4b6c-b989-cf0fc9cedacc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4a67e25f-29895270-mzh9z" podUID="a39db37e-e5c9-4b6c-b989-cf0fc9cedacc" Nov 03 14:30:03.663633 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:03.663604 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1ffaa0-29895270-xc62v" event={"ID":"f0590e2b-4337-4428-bc50-a2f1d54c289c","Type":"ContainerStarted","Data":"dfc27d54024c6c1326f34185939e9ab285d8c0292313acab0bdee25ff7c08784"} Nov 03 14:30:03.790371 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:03.790318 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895270-grzgd" podStartSLOduration=3.790304317 podStartE2EDuration="3.790304317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:30:03.786612277 +0000 UTC m=+1137.675401859" watchObservedRunningTime="2026-11-03 14:30:03.790304317 +0000 UTC m=+1137.679093891" Nov 03 14:30:04.251596 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:04.251559 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06a09272-2e46-4ee6-b035-74cea0c54feb" path="/var/lib/kubelet/pods/06a09272-2e46-4ee6-b035-74cea0c54feb/volumes" Nov 03 14:30:04.673660 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:04.673568 5060 generic.go:356] "Generic (PLEG): container finished" podID="73ee53fb-ff99-4bee-b2fa-c5e2c5874a68" containerID="83b0636c0019ba6eecb69ceeb0f9faabd6de52ce3fc76bc3d9a1a2cfda2baa1a" exitCode=60 Nov 03 14:30:04.674117 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:04.673647 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-gg8xh" event={"ID":"73ee53fb-ff99-4bee-b2fa-c5e2c5874a68","Type":"ContainerDied","Data":"83b0636c0019ba6eecb69ceeb0f9faabd6de52ce3fc76bc3d9a1a2cfda2baa1a"} Nov 03 14:30:04.674117 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:04.673746 5060 scope.go:122] "RemoveContainer" containerID="0aade806bc7f74d193c2a3f613a7b7a9c59e7d4e8d5eb83e4b67cd86ae84820d" Nov 03 14:30:04.674117 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:04.673987 5060 scope.go:122] "RemoveContainer" containerID="83b0636c0019ba6eecb69ceeb0f9faabd6de52ce3fc76bc3d9a1a2cfda2baa1a" Nov 03 14:30:04.674339 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:30:04.674310 5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895270-gg8xh_odh-ai-gateway-infra(73ee53fb-ff99-4bee-b2fa-c5e2c5874a68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-gg8xh" podUID="73ee53fb-ff99-4bee-b2fa-c5e2c5874a68" Nov 03 14:30:04.675966 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:04.675948 5060 generic.go:356] "Generic (PLEG): container finished" podID="59925bb5-ef87-4382-a542-9134ddb53466" containerID="fbf59c9c41778fecd38eca57fb9764de32edb3305bfbda389d93c4dbd777a634" exitCode=60 Nov 03 14:30:04.676060 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:04.676030 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895270-grzgd" event={"ID":"59925bb5-ef87-4382-a542-9134ddb53466","Type":"ContainerDied","Data":"fbf59c9c41778fecd38eca57fb9764de32edb3305bfbda389d93c4dbd777a634"} Nov 03 14:30:04.676214 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:04.676202 5060 scope.go:122] "RemoveContainer" containerID="fbf59c9c41778fecd38eca57fb9764de32edb3305bfbda389d93c4dbd777a634" Nov 03 14:30:04.676465 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:30:04.676447 5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-b4361f-29895270-grzgd_odh-ai-gateway-infra(59925bb5-ef87-4382-a542-9134ddb53466)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895270-grzgd" podUID="59925bb5-ef87-4382-a542-9134ddb53466" Nov 03 14:30:04.677739 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:04.677726 5060 scope.go:122] "RemoveContainer" containerID="bbf0408a7e33589d3567ce5ee6ce2721b91826da2410b8f2a0dec590fb21e5e5" Nov 03 14:30:04.677982 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:30:04.677966 5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-4a67e25f-29895270-mzh9z_odh-ai-gateway-infra(a39db37e-e5c9-4b6c-b989-cf0fc9cedacc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4a67e25f-29895270-mzh9z" podUID="a39db37e-e5c9-4b6c-b989-cf0fc9cedacc" Nov 03 14:30:04.679301 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:04.679270 5060 generic.go:356] "Generic (PLEG): container finished" podID="f0590e2b-4337-4428-bc50-a2f1d54c289c" containerID="dfc27d54024c6c1326f34185939e9ab285d8c0292313acab0bdee25ff7c08784" exitCode=60 Nov 03 14:30:04.679394 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:04.679321 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1ffaa0-29895270-xc62v" event={"ID":"f0590e2b-4337-4428-bc50-a2f1d54c289c","Type":"ContainerDied","Data":"dfc27d54024c6c1326f34185939e9ab285d8c0292313acab0bdee25ff7c08784"} Nov 03 14:30:04.679541 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:04.679528 5060 scope.go:122] "RemoveContainer" containerID="dfc27d54024c6c1326f34185939e9ab285d8c0292313acab0bdee25ff7c08784" Nov 03 14:30:04.679728 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:30:04.679712 5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-1ffaa0-29895270-xc62v_odh-ai-gateway-infra(f0590e2b-4337-4428-bc50-a2f1d54c289c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1ffaa0-29895270-xc62v" podUID="f0590e2b-4337-4428-bc50-a2f1d54c289c" Nov 03 14:30:04.687932 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:04.687910 5060 scope.go:122] "RemoveContainer" containerID="c1d7699880965a579036b9acaa2d1a4ce35ada6effb96b1fbfa25c19e0444a67" Nov 03 14:30:04.703504 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:04.703183 5060 scope.go:122] "RemoveContainer" containerID="c9a2d89ae3ca843a3e48215e236cc0a0c24a7defd2de6717f3abaaa59f85bf00" Nov 03 14:30:05.689764 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:05.689731 5060 scope.go:122] "RemoveContainer" containerID="83b0636c0019ba6eecb69ceeb0f9faabd6de52ce3fc76bc3d9a1a2cfda2baa1a" Nov 03 14:30:05.690325 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:30:05.690073 5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895270-gg8xh_odh-ai-gateway-infra(73ee53fb-ff99-4bee-b2fa-c5e2c5874a68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-gg8xh" podUID="73ee53fb-ff99-4bee-b2fa-c5e2c5874a68" Nov 03 14:30:05.693580 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:05.693561 5060 scope.go:122] "RemoveContainer" containerID="fbf59c9c41778fecd38eca57fb9764de32edb3305bfbda389d93c4dbd777a634" Nov 03 14:30:05.693846 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:30:05.693810 5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-b4361f-29895270-grzgd_odh-ai-gateway-infra(59925bb5-ef87-4382-a542-9134ddb53466)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895270-grzgd" podUID="59925bb5-ef87-4382-a542-9134ddb53466" Nov 03 14:30:05.695184 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:05.695167 5060 scope.go:122] "RemoveContainer" containerID="dfc27d54024c6c1326f34185939e9ab285d8c0292313acab0bdee25ff7c08784" Nov 03 14:30:05.695383 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:30:05.695367 5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-1ffaa0-29895270-xc62v_odh-ai-gateway-infra(f0590e2b-4337-4428-bc50-a2f1d54c289c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1ffaa0-29895270-xc62v" podUID="f0590e2b-4337-4428-bc50-a2f1d54c289c" Nov 03 14:30:06.576917 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:06.576872 5060 scope.go:122] "RemoveContainer" containerID="33675e8ffb2db0a802293f82bcf2fc5f086170de4e3d1c555f1b0217d1703352" Nov 03 14:30:09.449648 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:30:09.449602 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:30:10.191386 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:10.191350 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-4a67e25f/e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p" Nov 03 14:30:10.207100 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:10.207069 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-4a67e25f/e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p" Nov 03 14:30:10.543670 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:10.543638 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-1ffaa0/sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk"] Nov 03 14:30:10.549859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:10.549820 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk" Nov 03 14:30:10.569763 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:10.569733 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-1ffaa0\"/\"sub-test-model-1ffaa0-kserve-self-signed-certs\"" Nov 03 14:30:10.569939 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:10.569911 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-1ffaa0\"/\"openshift-service-ca.crt\"" Nov 03 14:30:10.570028 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:10.569979 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-1ffaa0\"/\"kube-root-ca.crt\"" Nov 03 14:30:10.570089 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:10.570079 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-1ffaa0\"/\"default-dockercfg-gk8b4\"" Nov 03 14:30:10.574208 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:10.574172 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-1ffaa0/sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk"] Nov 03 14:30:10.611279 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:10.611246 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49b53157-30cd-4fe2-8cca-a46e7dbec6f3-dshm\") pod \"sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk\" (UID: \"49b53157-30cd-4fe2-8cca-a46e7dbec6f3\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk" Nov 03 14:30:10.611447 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:10.611306 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49b53157-30cd-4fe2-8cca-a46e7dbec6f3-home\") pod \"sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk\" (UID: \"49b53157-30cd-4fe2-8cca-a46e7dbec6f3\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk" Nov 03 14:30:10.611447 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:10.611334 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2pzr\" (UniqueName: \"kubernetes.io/projected/49b53157-30cd-4fe2-8cca-a46e7dbec6f3-kube-api-access-k2pzr\") pod \"sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk\" (UID: \"49b53157-30cd-4fe2-8cca-a46e7dbec6f3\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk" Nov 03 14:30:10.611447 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:10.611376 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49b53157-30cd-4fe2-8cca-a46e7dbec6f3-tls-certs\") pod \"sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk\" (UID: \"49b53157-30cd-4fe2-8cca-a46e7dbec6f3\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk" Nov 03 14:30:10.611447 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:10.611441 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49b53157-30cd-4fe2-8cca-a46e7dbec6f3-model-cache\") pod \"sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk\" (UID: \"49b53157-30cd-4fe2-8cca-a46e7dbec6f3\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk" Nov 03 14:30:10.611702 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:10.611458 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49b53157-30cd-4fe2-8cca-a46e7dbec6f3-tmp-dir\") pod \"sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk\" (UID: \"49b53157-30cd-4fe2-8cca-a46e7dbec6f3\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk" Nov 03 14:30:10.712968 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:10.712918 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49b53157-30cd-4fe2-8cca-a46e7dbec6f3-tls-certs\") pod \"sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk\" (UID: \"49b53157-30cd-4fe2-8cca-a46e7dbec6f3\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk" Nov 03 14:30:10.713162 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:10.713051 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49b53157-30cd-4fe2-8cca-a46e7dbec6f3-model-cache\") pod \"sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk\" (UID: \"49b53157-30cd-4fe2-8cca-a46e7dbec6f3\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk" Nov 03 14:30:10.713162 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:10.713082 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49b53157-30cd-4fe2-8cca-a46e7dbec6f3-tmp-dir\") pod \"sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk\" (UID: \"49b53157-30cd-4fe2-8cca-a46e7dbec6f3\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk" Nov 03 14:30:10.713287 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:10.713256 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49b53157-30cd-4fe2-8cca-a46e7dbec6f3-dshm\") pod \"sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk\" (UID: \"49b53157-30cd-4fe2-8cca-a46e7dbec6f3\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk" Nov 03 14:30:10.713606 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:10.713573 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49b53157-30cd-4fe2-8cca-a46e7dbec6f3-model-cache\") pod \"sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk\" (UID: \"49b53157-30cd-4fe2-8cca-a46e7dbec6f3\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk" Nov 03 14:30:10.713732 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:10.713611 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49b53157-30cd-4fe2-8cca-a46e7dbec6f3-tmp-dir\") pod \"sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk\" (UID: \"49b53157-30cd-4fe2-8cca-a46e7dbec6f3\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk" Nov 03 14:30:10.713732 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:10.713670 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49b53157-30cd-4fe2-8cca-a46e7dbec6f3-home\") pod \"sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk\" (UID: \"49b53157-30cd-4fe2-8cca-a46e7dbec6f3\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk" Nov 03 14:30:10.713732 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:10.713724 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2pzr\" (UniqueName: \"kubernetes.io/projected/49b53157-30cd-4fe2-8cca-a46e7dbec6f3-kube-api-access-k2pzr\") pod \"sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk\" (UID: \"49b53157-30cd-4fe2-8cca-a46e7dbec6f3\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk" Nov 03 14:30:10.714042 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:10.714017 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49b53157-30cd-4fe2-8cca-a46e7dbec6f3-home\") pod \"sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk\" (UID: \"49b53157-30cd-4fe2-8cca-a46e7dbec6f3\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk" Nov 03 14:30:10.715692 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:10.715671 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49b53157-30cd-4fe2-8cca-a46e7dbec6f3-tls-certs\") pod \"sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk\" (UID: \"49b53157-30cd-4fe2-8cca-a46e7dbec6f3\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk" Nov 03 14:30:10.715918 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:10.715894 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49b53157-30cd-4fe2-8cca-a46e7dbec6f3-dshm\") pod \"sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk\" (UID: \"49b53157-30cd-4fe2-8cca-a46e7dbec6f3\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk" Nov 03 14:30:10.736963 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:10.736938 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2pzr\" (UniqueName: \"kubernetes.io/projected/49b53157-30cd-4fe2-8cca-a46e7dbec6f3-kube-api-access-k2pzr\") pod \"sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk\" (UID: \"49b53157-30cd-4fe2-8cca-a46e7dbec6f3\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk" Nov 03 14:30:10.770948 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:10.770916 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-4a67e25f/e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p" Nov 03 14:30:10.895540 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:10.895449 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk" Nov 03 14:30:11.226029 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:11.225999 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-1ffaa0/sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk"] Nov 03 14:30:11.227132 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:30:11.227107 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49b53157_30cd_4fe2_8cca_a46e7dbec6f3.slice/crio-30d6b36bef969ab95570d78e5061e37d589bfaa1090aac8f6681db7a8bd6476f WatchSource:0}: Error finding container 30d6b36bef969ab95570d78e5061e37d589bfaa1090aac8f6681db7a8bd6476f: Status 404 returned error can't find the container with id 30d6b36bef969ab95570d78e5061e37d589bfaa1090aac8f6681db7a8bd6476f Nov 03 14:30:11.768039 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:11.768001 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk" event={"ID":"49b53157-30cd-4fe2-8cca-a46e7dbec6f3","Type":"ContainerStarted","Data":"9d99b2f5d9ba9310223e64032bce1614ee059976537d1fee7924d31bff767b4b"} Nov 03 14:30:11.768039 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:11.768044 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk" event={"ID":"49b53157-30cd-4fe2-8cca-a46e7dbec6f3","Type":"ContainerStarted","Data":"30d6b36bef969ab95570d78e5061e37d589bfaa1090aac8f6681db7a8bd6476f"} Nov 03 14:30:11.817013 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:11.816948 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk" podStartSLOduration=1.816934386 podStartE2EDuration="1.816934386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:30:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:30:11.814292358 +0000 UTC m=+1145.703081942" watchObservedRunningTime="2026-11-03 14:30:11.816934386 +0000 UTC m=+1145.705723966" Nov 03 14:30:11.896968 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:11.896942 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk" Nov 03 14:30:14.242447 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:14.242414 5060 scope.go:122] "RemoveContainer" containerID="fbf59c9c41778fecd38eca57fb9764de32edb3305bfbda389d93c4dbd777a634" Nov 03 14:30:14.793779 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:14.793694 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895270-grzgd" event={"ID":"59925bb5-ef87-4382-a542-9134ddb53466","Type":"ContainerStarted","Data":"68c8b46cf54d4752e97d3b1ba4d01f136ac3263f94ccc1f17fa25c7947737e7a"} Nov 03 14:30:15.242190 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:15.242156 5060 scope.go:122] "RemoveContainer" containerID="83b0636c0019ba6eecb69ceeb0f9faabd6de52ce3fc76bc3d9a1a2cfda2baa1a" Nov 03 14:30:15.242372 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:15.242353 5060 scope.go:122] "RemoveContainer" containerID="dfc27d54024c6c1326f34185939e9ab285d8c0292313acab0bdee25ff7c08784" Nov 03 14:30:15.807503 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:15.807466 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-gg8xh" event={"ID":"73ee53fb-ff99-4bee-b2fa-c5e2c5874a68","Type":"ContainerStarted","Data":"7c2fa2af4c3f29cc0481080d0ec9d2a8c20dcb49416efdb35c6b9a4a22fa044e"} Nov 03 14:30:15.809377 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:15.809351 5060 generic.go:356] "Generic (PLEG): container finished" podID="59925bb5-ef87-4382-a542-9134ddb53466" containerID="68c8b46cf54d4752e97d3b1ba4d01f136ac3263f94ccc1f17fa25c7947737e7a" exitCode=60 Nov 03 14:30:15.809521 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:15.809432 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895270-grzgd" event={"ID":"59925bb5-ef87-4382-a542-9134ddb53466","Type":"ContainerDied","Data":"68c8b46cf54d4752e97d3b1ba4d01f136ac3263f94ccc1f17fa25c7947737e7a"} Nov 03 14:30:15.809521 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:15.809490 5060 scope.go:122] "RemoveContainer" containerID="fbf59c9c41778fecd38eca57fb9764de32edb3305bfbda389d93c4dbd777a634" Nov 03 14:30:15.809890 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:15.809869 5060 scope.go:122] "RemoveContainer" containerID="68c8b46cf54d4752e97d3b1ba4d01f136ac3263f94ccc1f17fa25c7947737e7a" Nov 03 14:30:15.810187 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:30:15.810155 5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-b4361f-29895270-grzgd_odh-ai-gateway-infra(59925bb5-ef87-4382-a542-9134ddb53466)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895270-grzgd" podUID="59925bb5-ef87-4382-a542-9134ddb53466" Nov 03 14:30:15.811563 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:15.811543 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1ffaa0-29895270-xc62v" event={"ID":"f0590e2b-4337-4428-bc50-a2f1d54c289c","Type":"ContainerStarted","Data":"6820ef8eefef23f8b6db799546513df9988427f92b685b8b0c6d52317e5f2b54"} Nov 03 14:30:15.851924 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:15.851892 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895270-grzgd"] Nov 03 14:30:16.821160 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:16.821128 5060 generic.go:356] "Generic (PLEG): container finished" podID="73ee53fb-ff99-4bee-b2fa-c5e2c5874a68" containerID="7c2fa2af4c3f29cc0481080d0ec9d2a8c20dcb49416efdb35c6b9a4a22fa044e" exitCode=60 Nov 03 14:30:16.821638 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:16.821210 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-gg8xh" event={"ID":"73ee53fb-ff99-4bee-b2fa-c5e2c5874a68","Type":"ContainerDied","Data":"7c2fa2af4c3f29cc0481080d0ec9d2a8c20dcb49416efdb35c6b9a4a22fa044e"} Nov 03 14:30:16.821638 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:16.821280 5060 scope.go:122] "RemoveContainer" containerID="83b0636c0019ba6eecb69ceeb0f9faabd6de52ce3fc76bc3d9a1a2cfda2baa1a" Nov 03 14:30:16.821748 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:16.821694 5060 scope.go:122] "RemoveContainer" containerID="7c2fa2af4c3f29cc0481080d0ec9d2a8c20dcb49416efdb35c6b9a4a22fa044e" Nov 03 14:30:16.822053 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:30:16.822030 5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895270-gg8xh_odh-ai-gateway-infra(73ee53fb-ff99-4bee-b2fa-c5e2c5874a68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-gg8xh" podUID="73ee53fb-ff99-4bee-b2fa-c5e2c5874a68" Nov 03 14:30:16.825773 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:16.825750 5060 generic.go:356] "Generic (PLEG): container finished" podID="f0590e2b-4337-4428-bc50-a2f1d54c289c" containerID="6820ef8eefef23f8b6db799546513df9988427f92b685b8b0c6d52317e5f2b54" exitCode=60 Nov 03 14:30:16.825889 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:16.825806 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1ffaa0-29895270-xc62v" event={"ID":"f0590e2b-4337-4428-bc50-a2f1d54c289c","Type":"ContainerDied","Data":"6820ef8eefef23f8b6db799546513df9988427f92b685b8b0c6d52317e5f2b54"} Nov 03 14:30:16.826192 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:16.826175 5060 scope.go:122] "RemoveContainer" containerID="6820ef8eefef23f8b6db799546513df9988427f92b685b8b0c6d52317e5f2b54" Nov 03 14:30:16.826424 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:30:16.826405 5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-1ffaa0-29895270-xc62v_odh-ai-gateway-infra(f0590e2b-4337-4428-bc50-a2f1d54c289c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1ffaa0-29895270-xc62v" podUID="f0590e2b-4337-4428-bc50-a2f1d54c289c" Nov 03 14:30:16.837549 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:16.837525 5060 scope.go:122] "RemoveContainer" containerID="dfc27d54024c6c1326f34185939e9ab285d8c0292313acab0bdee25ff7c08784" Nov 03 14:30:16.869482 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:16.869451 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895270-gg8xh"] Nov 03 14:30:16.955040 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:16.955010 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1ffaa0-29895270-xc62v"] Nov 03 14:30:16.992480 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:16.992454 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895270-grzgd" Nov 03 14:30:17.125336 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:17.125231 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59925bb5-ef87-4382-a542-9134ddb53466-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59925bb5-ef87-4382-a542-9134ddb53466-maas-api-service-ca\") pod \"59925bb5-ef87-4382-a542-9134ddb53466\" (UID: \"59925bb5-ef87-4382-a542-9134ddb53466\") " Nov 03 14:30:17.125504 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:17.125467 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59925bb5-ef87-4382-a542-9134ddb53466-kube-api-access-ddcjg\" (UniqueName: \"kubernetes.io/projected/59925bb5-ef87-4382-a542-9134ddb53466-kube-api-access-ddcjg\") pod \"59925bb5-ef87-4382-a542-9134ddb53466\" (UID: \"59925bb5-ef87-4382-a542-9134ddb53466\") " Nov 03 14:30:17.125607 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:17.125583 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59925bb5-ef87-4382-a542-9134ddb53466-maas-api-service-ca" pod "59925bb5-ef87-4382-a542-9134ddb53466" (UID: "59925bb5-ef87-4382-a542-9134ddb53466"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:30:17.125915 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:17.125895 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59925bb5-ef87-4382-a542-9134ddb53466-maas-api-service-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:30:17.127539 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:17.127512 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59925bb5-ef87-4382-a542-9134ddb53466-kube-api-access-ddcjg" pod "59925bb5-ef87-4382-a542-9134ddb53466" (UID: "59925bb5-ef87-4382-a542-9134ddb53466"). InnerVolumeSpecName "kube-api-access-ddcjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:30:17.227260 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:17.227215 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddcjg\" (UniqueName: \"kubernetes.io/projected/59925bb5-ef87-4382-a542-9134ddb53466-kube-api-access-ddcjg\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:30:17.242183 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:17.242149 5060 scope.go:122] "RemoveContainer" containerID="bbf0408a7e33589d3567ce5ee6ce2721b91826da2410b8f2a0dec590fb21e5e5" Nov 03 14:30:17.837819 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:17.837791 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895270-grzgd" Nov 03 14:30:17.838211 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:17.837784 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895270-grzgd" event={"ID":"59925bb5-ef87-4382-a542-9134ddb53466","Type":"ContainerDied","Data":"bd11a5678617033b7f08e1fcb49ffd59fac999d6de99dfc7132fa69655b4591d"} Nov 03 14:30:17.838211 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:17.837921 5060 scope.go:122] "RemoveContainer" containerID="68c8b46cf54d4752e97d3b1ba4d01f136ac3263f94ccc1f17fa25c7947737e7a" Nov 03 14:30:17.840349 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:17.840323 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4a67e25f-29895270-mzh9z" event={"ID":"a39db37e-e5c9-4b6c-b989-cf0fc9cedacc","Type":"ContainerStarted","Data":"8f6a32123a8e09459f02c55aed1f5209ea00d6dc5ff2f5bea5eb66478a708d9d"} Nov 03 14:30:17.981409 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:17.981360 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4a67e25f-29895270-mzh9z" podStartSLOduration=17.981345188 podStartE2EDuration="17.981345188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:30:17.937114439 +0000 UTC m=+1151.825904022" watchObservedRunningTime="2026-11-03 14:30:17.981345188 +0000 UTC m=+1151.870134780" Nov 03 14:30:17.981914 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:17.981894 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895270-grzgd"] Nov 03 14:30:17.992480 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:17.992453 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895270-grzgd"] Nov 03 14:30:18.043124 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:18.043090 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-gg8xh" Nov 03 14:30:18.093909 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:18.093847 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1ffaa0-29895270-xc62v" Nov 03 14:30:18.145815 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:18.144651 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73ee53fb-ff99-4bee-b2fa-c5e2c5874a68-kube-api-access-lddlc\" (UniqueName: \"kubernetes.io/projected/73ee53fb-ff99-4bee-b2fa-c5e2c5874a68-kube-api-access-lddlc\") pod \"73ee53fb-ff99-4bee-b2fa-c5e2c5874a68\" (UID: \"73ee53fb-ff99-4bee-b2fa-c5e2c5874a68\") " Nov 03 14:30:18.145815 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:18.144883 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73ee53fb-ff99-4bee-b2fa-c5e2c5874a68-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73ee53fb-ff99-4bee-b2fa-c5e2c5874a68-maas-api-service-ca\") pod \"73ee53fb-ff99-4bee-b2fa-c5e2c5874a68\" (UID: \"73ee53fb-ff99-4bee-b2fa-c5e2c5874a68\") " Nov 03 14:30:18.145815 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:18.145738 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73ee53fb-ff99-4bee-b2fa-c5e2c5874a68-maas-api-service-ca" pod "73ee53fb-ff99-4bee-b2fa-c5e2c5874a68" (UID: "73ee53fb-ff99-4bee-b2fa-c5e2c5874a68"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:30:18.148631 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:18.148602 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73ee53fb-ff99-4bee-b2fa-c5e2c5874a68-kube-api-access-lddlc" pod "73ee53fb-ff99-4bee-b2fa-c5e2c5874a68" (UID: "73ee53fb-ff99-4bee-b2fa-c5e2c5874a68"). InnerVolumeSpecName "kube-api-access-lddlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:30:18.245772 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:18.245742 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0590e2b-4337-4428-bc50-a2f1d54c289c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0590e2b-4337-4428-bc50-a2f1d54c289c-maas-api-service-ca\") pod \"f0590e2b-4337-4428-bc50-a2f1d54c289c\" (UID: \"f0590e2b-4337-4428-bc50-a2f1d54c289c\") " Nov 03 14:30:18.245959 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:18.245863 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0590e2b-4337-4428-bc50-a2f1d54c289c-kube-api-access-thldh\" (UniqueName: \"kubernetes.io/projected/f0590e2b-4337-4428-bc50-a2f1d54c289c-kube-api-access-thldh\") pod \"f0590e2b-4337-4428-bc50-a2f1d54c289c\" (UID: \"f0590e2b-4337-4428-bc50-a2f1d54c289c\") " Nov 03 14:30:18.246176 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:18.246152 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0590e2b-4337-4428-bc50-a2f1d54c289c-maas-api-service-ca" pod "f0590e2b-4337-4428-bc50-a2f1d54c289c" (UID: "f0590e2b-4337-4428-bc50-a2f1d54c289c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:30:18.246362 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:18.246338 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lddlc\" (UniqueName: \"kubernetes.io/projected/73ee53fb-ff99-4bee-b2fa-c5e2c5874a68-kube-api-access-lddlc\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:30:18.246437 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:18.246362 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0590e2b-4337-4428-bc50-a2f1d54c289c-maas-api-service-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:30:18.246437 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:18.246377 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73ee53fb-ff99-4bee-b2fa-c5e2c5874a68-maas-api-service-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:30:18.248097 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:18.248074 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0590e2b-4337-4428-bc50-a2f1d54c289c-kube-api-access-thldh" pod "f0590e2b-4337-4428-bc50-a2f1d54c289c" (UID: "f0590e2b-4337-4428-bc50-a2f1d54c289c"). InnerVolumeSpecName "kube-api-access-thldh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:30:18.249497 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:18.249476 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59925bb5-ef87-4382-a542-9134ddb53466" path="/var/lib/kubelet/pods/59925bb5-ef87-4382-a542-9134ddb53466/volumes" Nov 03 14:30:18.347961 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:18.347845 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thldh\" (UniqueName: \"kubernetes.io/projected/f0590e2b-4337-4428-bc50-a2f1d54c289c-kube-api-access-thldh\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:30:18.853688 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:18.853656 5060 generic.go:356] "Generic (PLEG): container finished" podID="a39db37e-e5c9-4b6c-b989-cf0fc9cedacc" containerID="8f6a32123a8e09459f02c55aed1f5209ea00d6dc5ff2f5bea5eb66478a708d9d" exitCode=60 Nov 03 14:30:18.854224 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:18.853726 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4a67e25f-29895270-mzh9z" event={"ID":"a39db37e-e5c9-4b6c-b989-cf0fc9cedacc","Type":"ContainerDied","Data":"8f6a32123a8e09459f02c55aed1f5209ea00d6dc5ff2f5bea5eb66478a708d9d"} Nov 03 14:30:18.854224 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:18.853794 5060 scope.go:122] "RemoveContainer" containerID="bbf0408a7e33589d3567ce5ee6ce2721b91826da2410b8f2a0dec590fb21e5e5" Nov 03 14:30:18.854224 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:18.854157 5060 scope.go:122] "RemoveContainer" containerID="8f6a32123a8e09459f02c55aed1f5209ea00d6dc5ff2f5bea5eb66478a708d9d" Nov 03 14:30:18.854433 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:30:18.854414 5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-4a67e25f-29895270-mzh9z_odh-ai-gateway-infra(a39db37e-e5c9-4b6c-b989-cf0fc9cedacc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4a67e25f-29895270-mzh9z" podUID="a39db37e-e5c9-4b6c-b989-cf0fc9cedacc" Nov 03 14:30:18.855141 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:18.855113 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1ffaa0-29895270-xc62v" Nov 03 14:30:18.855235 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:18.855152 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1ffaa0-29895270-xc62v" event={"ID":"f0590e2b-4337-4428-bc50-a2f1d54c289c","Type":"ContainerDied","Data":"db166d48aeb25ba8245bec66ccdf0b2e1a0d21704706b6794b2a81c0dd788450"} Nov 03 14:30:18.856705 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:18.856682 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-gg8xh" event={"ID":"73ee53fb-ff99-4bee-b2fa-c5e2c5874a68","Type":"ContainerDied","Data":"85ac8443a58392d03b685e58280c2ea170514d3b113f0ecb3a6998005d6e5277"} Nov 03 14:30:18.856818 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:18.856704 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-gg8xh" Nov 03 14:30:18.868305 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:18.868282 5060 scope.go:122] "RemoveContainer" containerID="6820ef8eefef23f8b6db799546513df9988427f92b685b8b0c6d52317e5f2b54" Nov 03 14:30:18.877873 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:18.877847 5060 scope.go:122] "RemoveContainer" containerID="7c2fa2af4c3f29cc0481080d0ec9d2a8c20dcb49416efdb35c6b9a4a22fa044e" Nov 03 14:30:18.938851 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:18.938807 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1ffaa0-29895270-xc62v"] Nov 03 14:30:18.947315 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:18.947268 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4a67e25f-29895270-mzh9z"] Nov 03 14:30:18.951112 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:18.951088 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1ffaa0-29895270-xc62v"] Nov 03 14:30:18.990071 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:18.990035 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895270-gg8xh"] Nov 03 14:30:18.994564 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:18.994534 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895270-gg8xh"] Nov 03 14:30:20.023214 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:20.023184 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4a67e25f-29895270-mzh9z" Nov 03 14:30:20.080923 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:20.080884 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a39db37e-e5c9-4b6c-b989-cf0fc9cedacc-kube-api-access-v7zww\" (UniqueName: \"kubernetes.io/projected/a39db37e-e5c9-4b6c-b989-cf0fc9cedacc-kube-api-access-v7zww\") pod \"a39db37e-e5c9-4b6c-b989-cf0fc9cedacc\" (UID: \"a39db37e-e5c9-4b6c-b989-cf0fc9cedacc\") " Nov 03 14:30:20.081454 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:20.081429 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a39db37e-e5c9-4b6c-b989-cf0fc9cedacc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a39db37e-e5c9-4b6c-b989-cf0fc9cedacc-maas-api-service-ca\") pod \"a39db37e-e5c9-4b6c-b989-cf0fc9cedacc\" (UID: \"a39db37e-e5c9-4b6c-b989-cf0fc9cedacc\") " Nov 03 14:30:20.081796 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:20.081768 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a39db37e-e5c9-4b6c-b989-cf0fc9cedacc-maas-api-service-ca" pod "a39db37e-e5c9-4b6c-b989-cf0fc9cedacc" (UID: "a39db37e-e5c9-4b6c-b989-cf0fc9cedacc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:30:20.082039 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:20.082015 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a39db37e-e5c9-4b6c-b989-cf0fc9cedacc-maas-api-service-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:30:20.083638 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:20.083613 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a39db37e-e5c9-4b6c-b989-cf0fc9cedacc-kube-api-access-v7zww" pod "a39db37e-e5c9-4b6c-b989-cf0fc9cedacc" (UID: "a39db37e-e5c9-4b6c-b989-cf0fc9cedacc"). InnerVolumeSpecName "kube-api-access-v7zww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:30:20.185942 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:20.185825 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7zww\" (UniqueName: \"kubernetes.io/projected/a39db37e-e5c9-4b6c-b989-cf0fc9cedacc-kube-api-access-v7zww\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:30:20.291859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:20.291362 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73ee53fb-ff99-4bee-b2fa-c5e2c5874a68" path="/var/lib/kubelet/pods/73ee53fb-ff99-4bee-b2fa-c5e2c5874a68/volumes" Nov 03 14:30:20.292063 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:20.292034 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0590e2b-4337-4428-bc50-a2f1d54c289c" path="/var/lib/kubelet/pods/f0590e2b-4337-4428-bc50-a2f1d54c289c/volumes" Nov 03 14:30:20.952860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:20.946038 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk" Nov 03 14:30:20.964861 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:20.962515 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4a67e25f-29895270-mzh9z" event={"ID":"a39db37e-e5c9-4b6c-b989-cf0fc9cedacc","Type":"ContainerDied","Data":"b17087ef59ed3a7fa8d706ec3347ca42062e853b148632c44da5485739f72c6a"} Nov 03 14:30:20.964861 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:20.962588 5060 scope.go:122] "RemoveContainer" containerID="8f6a32123a8e09459f02c55aed1f5209ea00d6dc5ff2f5bea5eb66478a708d9d" Nov 03 14:30:20.964861 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:20.962717 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4a67e25f-29895270-mzh9z" Nov 03 14:30:21.061882 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:21.058211 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4a67e25f-29895270-mzh9z"] Nov 03 14:30:21.112315 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:21.112283 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4a67e25f-29895270-mzh9z"] Nov 03 14:30:21.171858 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:21.170909 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-866507d3-kkqds"] Nov 03 14:30:21.193861 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:21.190482 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-866507d3-kkqds"] Nov 03 14:30:21.636860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:21.620201 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk" Nov 03 14:30:21.760981 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:21.760931 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-4a67e25f/e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p" podUID="0476cf91-73a3-4392-a1c7-1dc62e2a5b9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 14:30:22.289869 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:22.289430 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a39db37e-e5c9-4b6c-b989-cf0fc9cedacc" path="/var/lib/kubelet/pods/a39db37e-e5c9-4b6c-b989-cf0fc9cedacc/volumes" Nov 03 14:30:22.290348 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:22.290136 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3be4cf1-f542-46ac-92e4-61140d21c04c" path="/var/lib/kubelet/pods/a3be4cf1-f542-46ac-92e4-61140d21c04c/volumes" Nov 03 14:30:22.392037 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:22.392003 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk" Nov 03 14:30:31.294024 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:31.293980 5060 patch_prober.go:29] interesting pod/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 14:30:31.294576 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:31.294063 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" podUID="85bc4da4-ce30-4ea6-8a43-46f361b1f611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 14:30:31.437584 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:31.437503 5060 generic.go:356] "Generic (PLEG): container finished" podID="85bc4da4-ce30-4ea6-8a43-46f361b1f611" containerID="389992fe154fd4120fa10dc896e2d1cc98cca8050465774f6a5884a6d878a04a" exitCode=0 Nov 03 14:30:31.437727 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:31.437575 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" event={"ID":"85bc4da4-ce30-4ea6-8a43-46f361b1f611","Type":"ContainerDied","Data":"389992fe154fd4120fa10dc896e2d1cc98cca8050465774f6a5884a6d878a04a"} Nov 03 14:30:31.438106 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:31.438088 5060 scope.go:122] "RemoveContainer" containerID="389992fe154fd4120fa10dc896e2d1cc98cca8050465774f6a5884a6d878a04a" Nov 03 14:30:31.439487 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:31.439461 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:30:31.439562 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:31.439526 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:30:31.439620 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:31.439559 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:30:32.448057 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:32.448019 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" event={"ID":"85bc4da4-ce30-4ea6-8a43-46f361b1f611","Type":"ContainerStarted","Data":"d171ef79842e0d46166246058c4327b34f56564979fc5894735a6db68448d7e3"} Nov 03 14:30:32.949512 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:32.949469 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-2d0da2/sub-test-model-2d0da2-kserve-6f7f89db79-kffdh"] Nov 03 14:30:32.950971 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:32.950943 5060 cpu_manager.go:414] "containerMap: removing container" podUID="a39db37e-e5c9-4b6c-b989-cf0fc9cedacc" containerName="cleanup" Nov 03 14:30:32.951144 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:32.951130 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a39db37e-e5c9-4b6c-b989-cf0fc9cedacc" containerName="cleanup" Nov 03 14:30:32.951280 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:32.951270 5060 cpu_manager.go:414] "containerMap: removing container" podUID="f0590e2b-4337-4428-bc50-a2f1d54c289c" containerName="cleanup" Nov 03 14:30:32.951359 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:32.951349 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0590e2b-4337-4428-bc50-a2f1d54c289c" containerName="cleanup" Nov 03 14:30:32.951443 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:32.951434 5060 cpu_manager.go:414] "containerMap: removing container" podUID="73ee53fb-ff99-4bee-b2fa-c5e2c5874a68" containerName="cleanup" Nov 03 14:30:32.951509 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:32.951500 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73ee53fb-ff99-4bee-b2fa-c5e2c5874a68" containerName="cleanup" Nov 03 14:30:32.951580 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:32.951571 5060 cpu_manager.go:414] "containerMap: removing container" podUID="73ee53fb-ff99-4bee-b2fa-c5e2c5874a68" containerName="cleanup" Nov 03 14:30:32.951645 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:32.951636 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73ee53fb-ff99-4bee-b2fa-c5e2c5874a68" containerName="cleanup" Nov 03 14:30:32.951739 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:32.951730 5060 cpu_manager.go:414] "containerMap: removing container" podUID="59925bb5-ef87-4382-a542-9134ddb53466" containerName="cleanup" Nov 03 14:30:32.951816 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:32.951806 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59925bb5-ef87-4382-a542-9134ddb53466" containerName="cleanup" Nov 03 14:30:32.951925 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:32.951915 5060 cpu_manager.go:414] "containerMap: removing container" podUID="59925bb5-ef87-4382-a542-9134ddb53466" containerName="cleanup" Nov 03 14:30:32.951997 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:32.951988 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59925bb5-ef87-4382-a542-9134ddb53466" containerName="cleanup" Nov 03 14:30:32.952081 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:32.952072 5060 cpu_manager.go:414] "containerMap: removing container" podUID="f0590e2b-4337-4428-bc50-a2f1d54c289c" containerName="cleanup" Nov 03 14:30:32.952154 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:32.952146 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0590e2b-4337-4428-bc50-a2f1d54c289c" containerName="cleanup" Nov 03 14:30:32.952230 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:32.952221 5060 cpu_manager.go:414] "containerMap: removing container" podUID="73ee53fb-ff99-4bee-b2fa-c5e2c5874a68" containerName="cleanup" Nov 03 14:30:32.952316 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:32.952305 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73ee53fb-ff99-4bee-b2fa-c5e2c5874a68" containerName="cleanup" Nov 03 14:30:32.952444 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:32.952436 5060 cpu_manager.go:414] "containerMap: removing container" podUID="59925bb5-ef87-4382-a542-9134ddb53466" containerName="cleanup" Nov 03 14:30:32.952513 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:32.952504 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59925bb5-ef87-4382-a542-9134ddb53466" containerName="cleanup" Nov 03 14:30:32.952596 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:32.952587 5060 cpu_manager.go:414] "containerMap: removing container" podUID="a39db37e-e5c9-4b6c-b989-cf0fc9cedacc" containerName="cleanup" Nov 03 14:30:32.952659 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:32.952648 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a39db37e-e5c9-4b6c-b989-cf0fc9cedacc" containerName="cleanup" Nov 03 14:30:32.952995 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:32.952977 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="73ee53fb-ff99-4bee-b2fa-c5e2c5874a68" containerName="cleanup" Nov 03 14:30:32.953119 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:32.953109 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="73ee53fb-ff99-4bee-b2fa-c5e2c5874a68" containerName="cleanup" Nov 03 14:30:32.953194 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:32.953185 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0590e2b-4337-4428-bc50-a2f1d54c289c" containerName="cleanup" Nov 03 14:30:32.953262 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:32.953254 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="59925bb5-ef87-4382-a542-9134ddb53466" containerName="cleanup" Nov 03 14:30:32.953336 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:32.953328 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="59925bb5-ef87-4382-a542-9134ddb53466" containerName="cleanup" Nov 03 14:30:32.953413 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:32.953405 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="59925bb5-ef87-4382-a542-9134ddb53466" containerName="cleanup" Nov 03 14:30:32.953480 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:32.953472 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a39db37e-e5c9-4b6c-b989-cf0fc9cedacc" containerName="cleanup" Nov 03 14:30:32.953560 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:32.953552 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0590e2b-4337-4428-bc50-a2f1d54c289c" containerName="cleanup" Nov 03 14:30:32.953625 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:32.953617 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a39db37e-e5c9-4b6c-b989-cf0fc9cedacc" containerName="cleanup" Nov 03 14:30:32.953691 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:32.953683 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0590e2b-4337-4428-bc50-a2f1d54c289c" containerName="cleanup" Nov 03 14:30:32.958714 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:32.958692 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-2d0da2/sub-test-model-2d0da2-kserve-6f7f89db79-kffdh" Nov 03 14:30:32.974652 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:32.974620 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-2d0da2/sub-test-model-2d0da2-kserve-6f7f89db79-kffdh"] Nov 03 14:30:32.978649 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:32.978624 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-2d0da2\"/\"openshift-service-ca.crt\"" Nov 03 14:30:32.978649 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:32.978624 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-2d0da2\"/\"kube-root-ca.crt\"" Nov 03 14:30:32.981653 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:32.981630 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-2d0da2\"/\"sub-test-model-2d0da2-kserve-self-signed-certs\"" Nov 03 14:30:32.981774 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:32.981696 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-2d0da2\"/\"default-dockercfg-fpknl\"" Nov 03 14:30:33.057518 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:33.057474 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dc0b682-756a-4736-9d3a-ab20195b4bcb-tls-certs\") pod \"sub-test-model-2d0da2-kserve-6f7f89db79-kffdh\" (UID: \"3dc0b682-756a-4736-9d3a-ab20195b4bcb\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/sub-test-model-2d0da2-kserve-6f7f89db79-kffdh" Nov 03 14:30:33.057726 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:33.057531 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dc0b682-756a-4736-9d3a-ab20195b4bcb-dshm\") pod \"sub-test-model-2d0da2-kserve-6f7f89db79-kffdh\" (UID: \"3dc0b682-756a-4736-9d3a-ab20195b4bcb\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/sub-test-model-2d0da2-kserve-6f7f89db79-kffdh" Nov 03 14:30:33.057726 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:33.057649 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5xtp\" (UniqueName: \"kubernetes.io/projected/3dc0b682-756a-4736-9d3a-ab20195b4bcb-kube-api-access-m5xtp\") pod \"sub-test-model-2d0da2-kserve-6f7f89db79-kffdh\" (UID: \"3dc0b682-756a-4736-9d3a-ab20195b4bcb\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/sub-test-model-2d0da2-kserve-6f7f89db79-kffdh" Nov 03 14:30:33.057726 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:33.057687 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dc0b682-756a-4736-9d3a-ab20195b4bcb-home\") pod \"sub-test-model-2d0da2-kserve-6f7f89db79-kffdh\" (UID: \"3dc0b682-756a-4736-9d3a-ab20195b4bcb\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/sub-test-model-2d0da2-kserve-6f7f89db79-kffdh" Nov 03 14:30:33.057957 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:33.057766 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dc0b682-756a-4736-9d3a-ab20195b4bcb-model-cache\") pod \"sub-test-model-2d0da2-kserve-6f7f89db79-kffdh\" (UID: \"3dc0b682-756a-4736-9d3a-ab20195b4bcb\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/sub-test-model-2d0da2-kserve-6f7f89db79-kffdh" Nov 03 14:30:33.057957 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:33.057806 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dc0b682-756a-4736-9d3a-ab20195b4bcb-tmp-dir\") pod \"sub-test-model-2d0da2-kserve-6f7f89db79-kffdh\" (UID: \"3dc0b682-756a-4736-9d3a-ab20195b4bcb\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/sub-test-model-2d0da2-kserve-6f7f89db79-kffdh" Nov 03 14:30:33.159255 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:33.158813 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5xtp\" (UniqueName: \"kubernetes.io/projected/3dc0b682-756a-4736-9d3a-ab20195b4bcb-kube-api-access-m5xtp\") pod \"sub-test-model-2d0da2-kserve-6f7f89db79-kffdh\" (UID: \"3dc0b682-756a-4736-9d3a-ab20195b4bcb\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/sub-test-model-2d0da2-kserve-6f7f89db79-kffdh" Nov 03 14:30:33.159255 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:33.158886 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dc0b682-756a-4736-9d3a-ab20195b4bcb-home\") pod \"sub-test-model-2d0da2-kserve-6f7f89db79-kffdh\" (UID: \"3dc0b682-756a-4736-9d3a-ab20195b4bcb\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/sub-test-model-2d0da2-kserve-6f7f89db79-kffdh" Nov 03 14:30:33.159255 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:33.158970 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dc0b682-756a-4736-9d3a-ab20195b4bcb-model-cache\") pod \"sub-test-model-2d0da2-kserve-6f7f89db79-kffdh\" (UID: \"3dc0b682-756a-4736-9d3a-ab20195b4bcb\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/sub-test-model-2d0da2-kserve-6f7f89db79-kffdh" Nov 03 14:30:33.159255 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:33.159004 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dc0b682-756a-4736-9d3a-ab20195b4bcb-tmp-dir\") pod \"sub-test-model-2d0da2-kserve-6f7f89db79-kffdh\" (UID: \"3dc0b682-756a-4736-9d3a-ab20195b4bcb\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/sub-test-model-2d0da2-kserve-6f7f89db79-kffdh" Nov 03 14:30:33.159255 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:33.159095 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dc0b682-756a-4736-9d3a-ab20195b4bcb-tls-certs\") pod \"sub-test-model-2d0da2-kserve-6f7f89db79-kffdh\" (UID: \"3dc0b682-756a-4736-9d3a-ab20195b4bcb\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/sub-test-model-2d0da2-kserve-6f7f89db79-kffdh" Nov 03 14:30:33.159255 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:33.159129 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dc0b682-756a-4736-9d3a-ab20195b4bcb-dshm\") pod \"sub-test-model-2d0da2-kserve-6f7f89db79-kffdh\" (UID: \"3dc0b682-756a-4736-9d3a-ab20195b4bcb\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/sub-test-model-2d0da2-kserve-6f7f89db79-kffdh" Nov 03 14:30:33.159718 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:33.159428 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dc0b682-756a-4736-9d3a-ab20195b4bcb-home\") pod \"sub-test-model-2d0da2-kserve-6f7f89db79-kffdh\" (UID: \"3dc0b682-756a-4736-9d3a-ab20195b4bcb\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/sub-test-model-2d0da2-kserve-6f7f89db79-kffdh" Nov 03 14:30:33.159780 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:33.159740 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dc0b682-756a-4736-9d3a-ab20195b4bcb-tmp-dir\") pod \"sub-test-model-2d0da2-kserve-6f7f89db79-kffdh\" (UID: \"3dc0b682-756a-4736-9d3a-ab20195b4bcb\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/sub-test-model-2d0da2-kserve-6f7f89db79-kffdh" Nov 03 14:30:33.159934 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:33.159909 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dc0b682-756a-4736-9d3a-ab20195b4bcb-model-cache\") pod \"sub-test-model-2d0da2-kserve-6f7f89db79-kffdh\" (UID: \"3dc0b682-756a-4736-9d3a-ab20195b4bcb\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/sub-test-model-2d0da2-kserve-6f7f89db79-kffdh" Nov 03 14:30:33.163033 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:33.163002 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dc0b682-756a-4736-9d3a-ab20195b4bcb-dshm\") pod \"sub-test-model-2d0da2-kserve-6f7f89db79-kffdh\" (UID: \"3dc0b682-756a-4736-9d3a-ab20195b4bcb\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/sub-test-model-2d0da2-kserve-6f7f89db79-kffdh" Nov 03 14:30:33.163156 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:33.163000 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dc0b682-756a-4736-9d3a-ab20195b4bcb-tls-certs\") pod \"sub-test-model-2d0da2-kserve-6f7f89db79-kffdh\" (UID: \"3dc0b682-756a-4736-9d3a-ab20195b4bcb\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/sub-test-model-2d0da2-kserve-6f7f89db79-kffdh" Nov 03 14:30:33.201904 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:33.193868 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5xtp\" (UniqueName: \"kubernetes.io/projected/3dc0b682-756a-4736-9d3a-ab20195b4bcb-kube-api-access-m5xtp\") pod \"sub-test-model-2d0da2-kserve-6f7f89db79-kffdh\" (UID: \"3dc0b682-756a-4736-9d3a-ab20195b4bcb\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/sub-test-model-2d0da2-kserve-6f7f89db79-kffdh" Nov 03 14:30:33.278858 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:33.278553 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-2d0da2/sub-test-model-2d0da2-kserve-6f7f89db79-kffdh" Nov 03 14:30:33.445688 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:33.444168 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" Nov 03 14:30:33.445688 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:33.444289 5060 patch_prober.go:29] interesting pod/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 14:30:33.445688 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:33.444331 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" podUID="85bc4da4-ce30-4ea6-8a43-46f361b1f611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 14:30:33.638680 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:33.638646 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-2d0da2/sub-test-model-2d0da2-kserve-6f7f89db79-kffdh"] Nov 03 14:30:33.644469 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:30:33.644420 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3dc0b682_756a_4736_9d3a_ab20195b4bcb.slice/crio-e2a8b5592ab8c638035c1f317018983df28b1371365cd55bbebdfa7eb7a624b4 WatchSource:0}: Error finding container e2a8b5592ab8c638035c1f317018983df28b1371365cd55bbebdfa7eb7a624b4: Status 404 returned error can't find the container with id e2a8b5592ab8c638035c1f317018983df28b1371365cd55bbebdfa7eb7a624b4 Nov 03 14:30:34.443253 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:34.443221 5060 patch_prober.go:29] interesting pod/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 14:30:34.443416 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:34.443273 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" podUID="85bc4da4-ce30-4ea6-8a43-46f361b1f611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 14:30:34.464468 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:34.464424 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-2d0da2/sub-test-model-2d0da2-kserve-6f7f89db79-kffdh" event={"ID":"3dc0b682-756a-4736-9d3a-ab20195b4bcb","Type":"ContainerStarted","Data":"d0076503938c1f199853bd33c586b9bce5b58721880edcc552d3062bd2f3c706"} Nov 03 14:30:34.464646 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:34.464479 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-2d0da2/sub-test-model-2d0da2-kserve-6f7f89db79-kffdh" event={"ID":"3dc0b682-756a-4736-9d3a-ab20195b4bcb","Type":"ContainerStarted","Data":"e2a8b5592ab8c638035c1f317018983df28b1371365cd55bbebdfa7eb7a624b4"} Nov 03 14:30:34.525858 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:34.522141 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-2d0da2/sub-test-model-2d0da2-kserve-6f7f89db79-kffdh" podStartSLOduration=2.522120214 podStartE2EDuration="2.522120214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:30:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:30:34.508295454 +0000 UTC m=+1168.397085033" watchObservedRunningTime="2026-11-03 14:30:34.522120214 +0000 UTC m=+1168.410909797" Nov 03 14:30:35.279704 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:35.279671 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-2d0da2/sub-test-model-2d0da2-kserve-6f7f89db79-kffdh" Nov 03 14:30:35.443342 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:35.443303 5060 patch_prober.go:29] interesting pod/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 14:30:35.446146 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:35.444104 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" podUID="85bc4da4-ce30-4ea6-8a43-46f361b1f611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 14:30:35.472974 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:35.472942 5060 generic.go:356] "Generic (PLEG): container finished" podID="baf0a8cf-ee1d-4897-9b75-d28b2023c299" containerID="b38488471741ee7d8d82c8213d498089d51d8e7cb0fbbca56d38d46eb760f369" exitCode=0 Nov 03 14:30:35.473148 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:35.473015 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-shp6t-59fn9" event={"ID":"baf0a8cf-ee1d-4897-9b75-d28b2023c299","Type":"ContainerDied","Data":"b38488471741ee7d8d82c8213d498089d51d8e7cb0fbbca56d38d46eb760f369"} Nov 03 14:30:36.443351 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:36.443316 5060 patch_prober.go:29] interesting pod/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 14:30:36.443729 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:36.443375 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" podUID="85bc4da4-ce30-4ea6-8a43-46f361b1f611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 14:30:36.733458 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:36.733374 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-shp6t-59fn9" Nov 03 14:30:36.838215 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:36.838167 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baf0a8cf-ee1d-4897-9b75-d28b2023c299-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/baf0a8cf-ee1d-4897-9b75-d28b2023c299-archives-path\") pod \"baf0a8cf-ee1d-4897-9b75-d28b2023c299\" (UID: \"baf0a8cf-ee1d-4897-9b75-d28b2023c299\") " Nov 03 14:30:36.838416 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:36.838233 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baf0a8cf-ee1d-4897-9b75-d28b2023c299-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/baf0a8cf-ee1d-4897-9b75-d28b2023c299-trusted-ca-bundle\") pod \"baf0a8cf-ee1d-4897-9b75-d28b2023c299\" (UID: \"baf0a8cf-ee1d-4897-9b75-d28b2023c299\") " Nov 03 14:30:36.838416 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:36.838282 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baf0a8cf-ee1d-4897-9b75-d28b2023c299-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/baf0a8cf-ee1d-4897-9b75-d28b2023c299-service-ca-bundle\") pod \"baf0a8cf-ee1d-4897-9b75-d28b2023c299\" (UID: \"baf0a8cf-ee1d-4897-9b75-d28b2023c299\") " Nov 03 14:30:36.838416 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:36.838338 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baf0a8cf-ee1d-4897-9b75-d28b2023c299-kube-api-access-l5skb\" (UniqueName: \"kubernetes.io/projected/baf0a8cf-ee1d-4897-9b75-d28b2023c299-kube-api-access-l5skb\") pod \"baf0a8cf-ee1d-4897-9b75-d28b2023c299\" (UID: \"baf0a8cf-ee1d-4897-9b75-d28b2023c299\") " Nov 03 14:30:36.839874 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:36.839713 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baf0a8cf-ee1d-4897-9b75-d28b2023c299-trusted-ca-bundle" pod "baf0a8cf-ee1d-4897-9b75-d28b2023c299" (UID: "baf0a8cf-ee1d-4897-9b75-d28b2023c299"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:30:36.840038 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:36.839915 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf0a8cf-ee1d-4897-9b75-d28b2023c299-archives-path" pod "baf0a8cf-ee1d-4897-9b75-d28b2023c299" (UID: "baf0a8cf-ee1d-4897-9b75-d28b2023c299"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:36.840297 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:36.840272 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baf0a8cf-ee1d-4897-9b75-d28b2023c299-service-ca-bundle" pod "baf0a8cf-ee1d-4897-9b75-d28b2023c299" (UID: "baf0a8cf-ee1d-4897-9b75-d28b2023c299"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:30:36.841675 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:36.841650 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baf0a8cf-ee1d-4897-9b75-d28b2023c299-kube-api-access-l5skb" pod "baf0a8cf-ee1d-4897-9b75-d28b2023c299" (UID: "baf0a8cf-ee1d-4897-9b75-d28b2023c299"). InnerVolumeSpecName "kube-api-access-l5skb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:30:36.940325 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:36.940286 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5skb\" (UniqueName: \"kubernetes.io/projected/baf0a8cf-ee1d-4897-9b75-d28b2023c299-kube-api-access-l5skb\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:30:36.940325 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:36.940324 5060 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/baf0a8cf-ee1d-4897-9b75-d28b2023c299-archives-path\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:30:36.940325 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:36.940335 5060 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/baf0a8cf-ee1d-4897-9b75-d28b2023c299-trusted-ca-bundle\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:30:36.940566 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:36.940343 5060 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/baf0a8cf-ee1d-4897-9b75-d28b2023c299-service-ca-bundle\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:30:37.443274 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:37.443234 5060 patch_prober.go:29] interesting pod/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 14:30:37.443464 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:37.443302 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" podUID="85bc4da4-ce30-4ea6-8a43-46f361b1f611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 14:30:37.493995 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:37.493960 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-shp6t-59fn9" event={"ID":"baf0a8cf-ee1d-4897-9b75-d28b2023c299","Type":"ContainerDied","Data":"1e413ce5274345f3fd85e3ab04c27940caff6cd5815b24abdc2acf2871f5d438"} Nov 03 14:30:37.494258 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:37.494015 5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e413ce5274345f3fd85e3ab04c27940caff6cd5815b24abdc2acf2871f5d438" Nov 03 14:30:37.494258 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:37.494105 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-shp6t-59fn9" Nov 03 14:30:38.808404 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:38.808356 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" Nov 03 14:30:38.810345 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:38.810187 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" Nov 03 14:30:38.811990 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:38.811918 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" Nov 03 14:30:39.544854 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:30:39.544102 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:30:43.281076 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:43.280309 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-2d0da2/sub-test-model-2d0da2-kserve-6f7f89db79-kffdh" Nov 03 14:30:43.298473 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:43.298443 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-2d0da2/sub-test-model-2d0da2-kserve-6f7f89db79-kffdh" Nov 03 14:30:43.386642 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:43.386609 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq"] Nov 03 14:30:43.387918 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:43.387878 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" podUID="85bc4da4-ce30-4ea6-8a43-46f361b1f611" containerName="istio-proxy" containerID="cri-o://d171ef79842e0d46166246058c4327b34f56564979fc5894735a6db68448d7e3" gracePeriod=30 Nov 03 14:30:43.621333 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:43.621248 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-2d0da2/sub-test-model-2d0da2-kserve-6f7f89db79-kffdh" Nov 03 14:30:48.099314 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.099273 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a35a5574-sxdjh"] Nov 03 14:30:48.099996 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.099973 5060 cpu_manager.go:414] "containerMap: removing container" podUID="f0590e2b-4337-4428-bc50-a2f1d54c289c" containerName="cleanup" Nov 03 14:30:48.100108 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.099997 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0590e2b-4337-4428-bc50-a2f1d54c289c" containerName="cleanup" Nov 03 14:30:48.100108 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.100019 5060 cpu_manager.go:414] "containerMap: removing container" podUID="baf0a8cf-ee1d-4897-9b75-d28b2023c299" containerName="insights-gathering" Nov 03 14:30:48.100108 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.100027 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baf0a8cf-ee1d-4897-9b75-d28b2023c299" containerName="insights-gathering" Nov 03 14:30:48.100108 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.100061 5060 cpu_manager.go:414] "containerMap: removing container" podUID="a39db37e-e5c9-4b6c-b989-cf0fc9cedacc" containerName="cleanup" Nov 03 14:30:48.100108 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.100068 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a39db37e-e5c9-4b6c-b989-cf0fc9cedacc" containerName="cleanup" Nov 03 14:30:48.100367 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.100227 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="73ee53fb-ff99-4bee-b2fa-c5e2c5874a68" containerName="cleanup" Nov 03 14:30:48.100367 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.100245 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a39db37e-e5c9-4b6c-b989-cf0fc9cedacc" containerName="cleanup" Nov 03 14:30:48.100367 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.100268 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="baf0a8cf-ee1d-4897-9b75-d28b2023c299" containerName="insights-gathering" Nov 03 14:30:48.105146 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.105122 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a35a5574-sxdjh" Nov 03 14:30:48.127593 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.127561 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a35a5574-sxdjh"] Nov 03 14:30:48.136188 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.136159 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\"" Nov 03 14:30:48.136405 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.136220 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:30:48.136500 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.136359 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:30:48.136583 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.136397 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:30:48.142828 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.142796 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7319-1c0cee"] Nov 03 14:30:48.147619 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.147596 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7319-1c0cee" Nov 03 14:30:48.162079 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.162048 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7319-1c0cee"] Nov 03 14:30:48.196627 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.196593 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crwr2\" (UniqueName: \"kubernetes.io/projected/9c4ff8e1-33dc-4e0b-a735-9b64513bd315-kube-api-access-crwr2\") pod \"test-curl-7319-1c0cee\" (UID: \"9c4ff8e1-33dc-4e0b-a735-9b64513bd315\") " pod="opendatahub/test-curl-7319-1c0cee" Nov 03 14:30:48.196829 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.196657 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dea710fa-d5e0-463e-adff-8d708346f953-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a35a5574-sxdjh\" (UID: \"dea710fa-d5e0-463e-adff-8d708346f953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a35a5574-sxdjh" Nov 03 14:30:48.196829 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.196779 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dea710fa-d5e0-463e-adff-8d708346f953-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a35a5574-sxdjh\" (UID: \"dea710fa-d5e0-463e-adff-8d708346f953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a35a5574-sxdjh" Nov 03 14:30:48.196971 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.196824 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmk7k\" (UniqueName: \"kubernetes.io/projected/dea710fa-d5e0-463e-adff-8d708346f953-kube-api-access-gmk7k\") pod \"maas-api-revoke-sub-a35a5574-sxdjh\" (UID: \"dea710fa-d5e0-463e-adff-8d708346f953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a35a5574-sxdjh" Nov 03 14:30:48.224391 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.224361 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-9sm86\"" Nov 03 14:30:48.298773 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.298726 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crwr2\" (UniqueName: \"kubernetes.io/projected/9c4ff8e1-33dc-4e0b-a735-9b64513bd315-kube-api-access-crwr2\") pod \"test-curl-7319-1c0cee\" (UID: \"9c4ff8e1-33dc-4e0b-a735-9b64513bd315\") " pod="opendatahub/test-curl-7319-1c0cee" Nov 03 14:30:48.299125 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.299102 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dea710fa-d5e0-463e-adff-8d708346f953-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a35a5574-sxdjh\" (UID: \"dea710fa-d5e0-463e-adff-8d708346f953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a35a5574-sxdjh" Nov 03 14:30:48.299323 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.299307 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dea710fa-d5e0-463e-adff-8d708346f953-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a35a5574-sxdjh\" (UID: \"dea710fa-d5e0-463e-adff-8d708346f953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a35a5574-sxdjh" Nov 03 14:30:48.299452 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.299426 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmk7k\" (UniqueName: \"kubernetes.io/projected/dea710fa-d5e0-463e-adff-8d708346f953-kube-api-access-gmk7k\") pod \"maas-api-revoke-sub-a35a5574-sxdjh\" (UID: \"dea710fa-d5e0-463e-adff-8d708346f953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a35a5574-sxdjh" Nov 03 14:30:48.300024 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.300001 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dea710fa-d5e0-463e-adff-8d708346f953-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a35a5574-sxdjh\" (UID: \"dea710fa-d5e0-463e-adff-8d708346f953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a35a5574-sxdjh" Nov 03 14:30:48.302540 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.302512 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dea710fa-d5e0-463e-adff-8d708346f953-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a35a5574-sxdjh\" (UID: \"dea710fa-d5e0-463e-adff-8d708346f953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a35a5574-sxdjh" Nov 03 14:30:48.321323 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.321291 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmk7k\" (UniqueName: \"kubernetes.io/projected/dea710fa-d5e0-463e-adff-8d708346f953-kube-api-access-gmk7k\") pod \"maas-api-revoke-sub-a35a5574-sxdjh\" (UID: \"dea710fa-d5e0-463e-adff-8d708346f953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a35a5574-sxdjh" Nov 03 14:30:48.321522 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.321505 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crwr2\" (UniqueName: \"kubernetes.io/projected/9c4ff8e1-33dc-4e0b-a735-9b64513bd315-kube-api-access-crwr2\") pod \"test-curl-7319-1c0cee\" (UID: \"9c4ff8e1-33dc-4e0b-a735-9b64513bd315\") " pod="opendatahub/test-curl-7319-1c0cee" Nov 03 14:30:48.419605 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.419519 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a35a5574-sxdjh" Nov 03 14:30:48.463726 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.463699 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7319-1c0cee" Nov 03 14:30:48.660750 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.660557 5060 generic.go:356] "Generic (PLEG): container finished" podID="85bc4da4-ce30-4ea6-8a43-46f361b1f611" containerID="d171ef79842e0d46166246058c4327b34f56564979fc5894735a6db68448d7e3" exitCode=0 Nov 03 14:30:48.661172 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.660971 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" event={"ID":"85bc4da4-ce30-4ea6-8a43-46f361b1f611","Type":"ContainerDied","Data":"d171ef79842e0d46166246058c4327b34f56564979fc5894735a6db68448d7e3"} Nov 03 14:30:48.661172 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.661064 5060 scope.go:122] "RemoveContainer" containerID="389992fe154fd4120fa10dc896e2d1cc98cca8050465774f6a5884a6d878a04a" Nov 03 14:30:48.773478 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.773444 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" Nov 03 14:30:48.787796 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.787763 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a35a5574-sxdjh"] Nov 03 14:30:48.793275 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:30:48.793245 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddea710fa_d5e0_463e_adff_8d708346f953.slice/crio-170d81e59989496ff8d777514351a45b97cb3463365ecdcbe3d98a4ef997e43a WatchSource:0}: Error finding container 170d81e59989496ff8d777514351a45b97cb3463365ecdcbe3d98a4ef997e43a: Status 404 returned error can't find the container with id 170d81e59989496ff8d777514351a45b97cb3463365ecdcbe3d98a4ef997e43a Nov 03 14:30:48.801570 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.801544 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7319-1c0cee"] Nov 03 14:30:48.808289 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.808257 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85bc4da4-ce30-4ea6-8a43-46f361b1f611-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85bc4da4-ce30-4ea6-8a43-46f361b1f611-istio-podinfo\") pod \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\" (UID: \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\") " Nov 03 14:30:48.808410 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.808329 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85bc4da4-ce30-4ea6-8a43-46f361b1f611-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bc4da4-ce30-4ea6-8a43-46f361b1f611-credential-socket\") pod \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\" (UID: \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\") " Nov 03 14:30:48.808410 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.808375 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85bc4da4-ce30-4ea6-8a43-46f361b1f611-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85bc4da4-ce30-4ea6-8a43-46f361b1f611-workload-certs\") pod \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\" (UID: \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\") " Nov 03 14:30:48.808410 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.808399 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85bc4da4-ce30-4ea6-8a43-46f361b1f611-istio-token\" (UniqueName: \"kubernetes.io/projected/85bc4da4-ce30-4ea6-8a43-46f361b1f611-istio-token\") pod \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\" (UID: \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\") " Nov 03 14:30:48.808561 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.808419 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85bc4da4-ce30-4ea6-8a43-46f361b1f611-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bc4da4-ce30-4ea6-8a43-46f361b1f611-workload-socket\") pod \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\" (UID: \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\") " Nov 03 14:30:48.808561 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.808453 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85bc4da4-ce30-4ea6-8a43-46f361b1f611-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85bc4da4-ce30-4ea6-8a43-46f361b1f611-istio-data\") pod \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\" (UID: \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\") " Nov 03 14:30:48.808561 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.808493 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85bc4da4-ce30-4ea6-8a43-46f361b1f611-kube-api-access-z7m5d\" (UniqueName: \"kubernetes.io/projected/85bc4da4-ce30-4ea6-8a43-46f361b1f611-kube-api-access-z7m5d\") pod \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\" (UID: \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\") " Nov 03 14:30:48.808561 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.808548 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85bc4da4-ce30-4ea6-8a43-46f361b1f611-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85bc4da4-ce30-4ea6-8a43-46f361b1f611-istiod-ca-cert\") pod \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\" (UID: \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\") " Nov 03 14:30:48.808770 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.808565 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85bc4da4-ce30-4ea6-8a43-46f361b1f611-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85bc4da4-ce30-4ea6-8a43-46f361b1f611-istio-envoy\") pod \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\" (UID: \"85bc4da4-ce30-4ea6-8a43-46f361b1f611\") " Nov 03 14:30:48.809409 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.809382 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85bc4da4-ce30-4ea6-8a43-46f361b1f611-istiod-ca-cert" pod "85bc4da4-ce30-4ea6-8a43-46f361b1f611" (UID: "85bc4da4-ce30-4ea6-8a43-46f361b1f611"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:30:48.809409 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.809392 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85bc4da4-ce30-4ea6-8a43-46f361b1f611-istio-data" pod "85bc4da4-ce30-4ea6-8a43-46f361b1f611" (UID: "85bc4da4-ce30-4ea6-8a43-46f361b1f611"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:48.809638 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.809619 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85bc4da4-ce30-4ea6-8a43-46f361b1f611-credential-socket" pod "85bc4da4-ce30-4ea6-8a43-46f361b1f611" (UID: "85bc4da4-ce30-4ea6-8a43-46f361b1f611"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:48.809717 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.809636 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85bc4da4-ce30-4ea6-8a43-46f361b1f611-workload-socket" pod "85bc4da4-ce30-4ea6-8a43-46f361b1f611" (UID: "85bc4da4-ce30-4ea6-8a43-46f361b1f611"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:48.809929 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.809904 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85bc4da4-ce30-4ea6-8a43-46f361b1f611-workload-certs" pod "85bc4da4-ce30-4ea6-8a43-46f361b1f611" (UID: "85bc4da4-ce30-4ea6-8a43-46f361b1f611"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:48.810699 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:30:48.810674 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c4ff8e1_33dc_4e0b_a735_9b64513bd315.slice/crio-2d0bd8adf00214d5492fa9a9b9c8fe45fcb445ba7ade0167d7847e6bb8d407a9 WatchSource:0}: Error finding container 2d0bd8adf00214d5492fa9a9b9c8fe45fcb445ba7ade0167d7847e6bb8d407a9: Status 404 returned error can't find the container with id 2d0bd8adf00214d5492fa9a9b9c8fe45fcb445ba7ade0167d7847e6bb8d407a9 Nov 03 14:30:48.811975 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.811775 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85bc4da4-ce30-4ea6-8a43-46f361b1f611-kube-api-access-z7m5d" pod "85bc4da4-ce30-4ea6-8a43-46f361b1f611" (UID: "85bc4da4-ce30-4ea6-8a43-46f361b1f611"). InnerVolumeSpecName "kube-api-access-z7m5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:30:48.812564 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.812547 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85bc4da4-ce30-4ea6-8a43-46f361b1f611-istio-podinfo" pod "85bc4da4-ce30-4ea6-8a43-46f361b1f611" (UID: "85bc4da4-ce30-4ea6-8a43-46f361b1f611"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:30:48.813074 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.813008 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85bc4da4-ce30-4ea6-8a43-46f361b1f611-istio-envoy" pod "85bc4da4-ce30-4ea6-8a43-46f361b1f611" (UID: "85bc4da4-ce30-4ea6-8a43-46f361b1f611"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:48.813815 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.813793 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85bc4da4-ce30-4ea6-8a43-46f361b1f611-istio-token" pod "85bc4da4-ce30-4ea6-8a43-46f361b1f611" (UID: "85bc4da4-ce30-4ea6-8a43-46f361b1f611"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:30:48.909759 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.909714 5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bc4da4-ce30-4ea6-8a43-46f361b1f611-credential-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:30:48.909759 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.909741 5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85bc4da4-ce30-4ea6-8a43-46f361b1f611-workload-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:30:48.909759 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.909750 5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85bc4da4-ce30-4ea6-8a43-46f361b1f611-istio-token\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:30:48.909759 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.909759 5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bc4da4-ce30-4ea6-8a43-46f361b1f611-workload-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:30:48.909759 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.909767 5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85bc4da4-ce30-4ea6-8a43-46f361b1f611-istio-data\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:30:48.910116 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.909777 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7m5d\" (UniqueName: \"kubernetes.io/projected/85bc4da4-ce30-4ea6-8a43-46f361b1f611-kube-api-access-z7m5d\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:30:48.910116 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.909785 5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85bc4da4-ce30-4ea6-8a43-46f361b1f611-istiod-ca-cert\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:30:48.910116 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.909793 5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85bc4da4-ce30-4ea6-8a43-46f361b1f611-istio-envoy\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:30:48.910116 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:48.909800 5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85bc4da4-ce30-4ea6-8a43-46f361b1f611-istio-podinfo\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:30:49.670763 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:49.670681 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" Nov 03 14:30:49.670763 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:49.670693 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq" event={"ID":"85bc4da4-ce30-4ea6-8a43-46f361b1f611","Type":"ContainerDied","Data":"b7ca6cc7d505f9f1ef7901d66281bb54d94a89f8ab1dcb7d2a3d603516b30f0f"} Nov 03 14:30:49.670763 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:49.670762 5060 scope.go:122] "RemoveContainer" containerID="d171ef79842e0d46166246058c4327b34f56564979fc5894735a6db68448d7e3" Nov 03 14:30:49.672621 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:49.672593 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7319-1c0cee" event={"ID":"9c4ff8e1-33dc-4e0b-a735-9b64513bd315","Type":"ContainerStarted","Data":"fa80cf8bf96e3a5c9b6affe08fddf7f8da7c118ac820819d09281745271a14de"} Nov 03 14:30:49.672745 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:49.672632 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7319-1c0cee" event={"ID":"9c4ff8e1-33dc-4e0b-a735-9b64513bd315","Type":"ContainerStarted","Data":"2d0bd8adf00214d5492fa9a9b9c8fe45fcb445ba7ade0167d7847e6bb8d407a9"} Nov 03 14:30:49.674413 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:49.674389 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a35a5574-sxdjh" event={"ID":"dea710fa-d5e0-463e-adff-8d708346f953","Type":"ContainerStarted","Data":"e3f004809bbde156e9f9fc103f8190afa70ca5983b1a3fe3464ee3dc91f04c09"} Nov 03 14:30:49.674527 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:49.674441 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a35a5574-sxdjh" event={"ID":"dea710fa-d5e0-463e-adff-8d708346f953","Type":"ContainerStarted","Data":"170d81e59989496ff8d777514351a45b97cb3463365ecdcbe3d98a4ef997e43a"} Nov 03 14:30:49.734751 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:49.734689 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a35a5574-sxdjh" podStartSLOduration=1.734671069 podStartE2EDuration="1.734671069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:30:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:30:49.731136423 +0000 UTC m=+1183.619926004" watchObservedRunningTime="2026-11-03 14:30:49.734671069 +0000 UTC m=+1183.623460653" Nov 03 14:30:49.775573 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:49.775529 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7319-1c0cee" podStartSLOduration=1.7755143389999999 podStartE2EDuration="1.775514339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:30:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:30:49.771589938 +0000 UTC m=+1183.660379519" watchObservedRunningTime="2026-11-03 14:30:49.775514339 +0000 UTC m=+1183.664303920" Nov 03 14:30:49.813453 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:49.813417 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq"] Nov 03 14:30:49.820346 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:49.820311 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-d608daf1-gw-openshift-default-cf68dbbf-kltrq"] Nov 03 14:30:50.251900 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:50.251865 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85bc4da4-ce30-4ea6-8a43-46f361b1f611" path="/var/lib/kubelet/pods/85bc4da4-ce30-4ea6-8a43-46f361b1f611/volumes" Nov 03 14:30:50.685761 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:50.685677 5060 generic.go:356] "Generic (PLEG): container finished" podID="dea710fa-d5e0-463e-adff-8d708346f953" containerID="e3f004809bbde156e9f9fc103f8190afa70ca5983b1a3fe3464ee3dc91f04c09" exitCode=0 Nov 03 14:30:50.686136 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:50.685754 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a35a5574-sxdjh" event={"ID":"dea710fa-d5e0-463e-adff-8d708346f953","Type":"ContainerDied","Data":"e3f004809bbde156e9f9fc103f8190afa70ca5983b1a3fe3464ee3dc91f04c09"} Nov 03 14:30:50.820010 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:50.819972 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7319-1c0cee"] Nov 03 14:30:50.825788 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:50.825750 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7319-1c0cee"] Nov 03 14:30:51.693666 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:51.693619 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7319-1c0cee" podUID="9c4ff8e1-33dc-4e0b-a735-9b64513bd315" containerName="test-curl-7319-1c0cee" containerID="cri-o://fa80cf8bf96e3a5c9b6affe08fddf7f8da7c118ac820819d09281745271a14de" gracePeriod=2 Nov 03 14:30:51.707814 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:30:51.707775 5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7319-1c0cee\" is forbidden: User \"system:node:ip-10-0-143-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-171.ec2.internal' and this object" podUID="9c4ff8e1-33dc-4e0b-a735-9b64513bd315" pod="opendatahub/test-curl-7319-1c0cee" Nov 03 14:30:51.868743 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:51.868712 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a35a5574-sxdjh" Nov 03 14:30:51.883081 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:30:51.883043 5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7319-1c0cee\" is forbidden: User \"system:node:ip-10-0-143-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-171.ec2.internal' and this object" podUID="9c4ff8e1-33dc-4e0b-a735-9b64513bd315" pod="opendatahub/test-curl-7319-1c0cee" Nov 03 14:30:51.959348 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:51.959250 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dea710fa-d5e0-463e-adff-8d708346f953-kube-api-access-gmk7k\" (UniqueName: \"kubernetes.io/projected/dea710fa-d5e0-463e-adff-8d708346f953-kube-api-access-gmk7k\") pod \"dea710fa-d5e0-463e-adff-8d708346f953\" (UID: \"dea710fa-d5e0-463e-adff-8d708346f953\") " Nov 03 14:30:51.959348 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:51.959333 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dea710fa-d5e0-463e-adff-8d708346f953-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dea710fa-d5e0-463e-adff-8d708346f953-maas-api-xks-ca\") pod \"dea710fa-d5e0-463e-adff-8d708346f953\" (UID: \"dea710fa-d5e0-463e-adff-8d708346f953\") " Nov 03 14:30:51.959567 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:51.959393 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dea710fa-d5e0-463e-adff-8d708346f953-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dea710fa-d5e0-463e-adff-8d708346f953-maas-api-openshift-ca\") pod \"dea710fa-d5e0-463e-adff-8d708346f953\" (UID: \"dea710fa-d5e0-463e-adff-8d708346f953\") " Nov 03 14:30:51.959800 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:51.959777 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dea710fa-d5e0-463e-adff-8d708346f953-maas-api-openshift-ca" pod "dea710fa-d5e0-463e-adff-8d708346f953" (UID: "dea710fa-d5e0-463e-adff-8d708346f953"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:30:51.961457 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:51.961417 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dea710fa-d5e0-463e-adff-8d708346f953-maas-api-xks-ca" pod "dea710fa-d5e0-463e-adff-8d708346f953" (UID: "dea710fa-d5e0-463e-adff-8d708346f953"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:30:51.961457 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:51.961433 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dea710fa-d5e0-463e-adff-8d708346f953-kube-api-access-gmk7k" pod "dea710fa-d5e0-463e-adff-8d708346f953" (UID: "dea710fa-d5e0-463e-adff-8d708346f953"). InnerVolumeSpecName "kube-api-access-gmk7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:30:52.041706 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:52.041669 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81c16996-v4vdl"] Nov 03 14:30:52.047155 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:52.047123 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81c16996-v4vdl"] Nov 03 14:30:52.061279 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:52.061248 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmk7k\" (UniqueName: \"kubernetes.io/projected/dea710fa-d5e0-463e-adff-8d708346f953-kube-api-access-gmk7k\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:30:52.061279 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:52.061273 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dea710fa-d5e0-463e-adff-8d708346f953-maas-api-xks-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:30:52.061450 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:52.061287 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dea710fa-d5e0-463e-adff-8d708346f953-maas-api-openshift-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:30:52.250564 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:52.250529 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf486f2f-1712-40b4-8af2-e8e5a9f8aaae" path="/var/lib/kubelet/pods/bf486f2f-1712-40b4-8af2-e8e5a9f8aaae/volumes" Nov 03 14:30:52.703595 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:52.703499 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a35a5574-sxdjh" Nov 03 14:30:52.703998 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:52.703519 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a35a5574-sxdjh" event={"ID":"dea710fa-d5e0-463e-adff-8d708346f953","Type":"ContainerDied","Data":"170d81e59989496ff8d777514351a45b97cb3463365ecdcbe3d98a4ef997e43a"} Nov 03 14:30:52.703998 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:52.703659 5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="170d81e59989496ff8d777514351a45b97cb3463365ecdcbe3d98a4ef997e43a" Nov 03 14:30:54.545859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:54.538003 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7319-1c0cee" Nov 03 14:30:54.620339 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:54.620293 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c4ff8e1-33dc-4e0b-a735-9b64513bd315-kube-api-access-crwr2\" (UniqueName: \"kubernetes.io/projected/9c4ff8e1-33dc-4e0b-a735-9b64513bd315-kube-api-access-crwr2\") pod \"9c4ff8e1-33dc-4e0b-a735-9b64513bd315\" (UID: \"9c4ff8e1-33dc-4e0b-a735-9b64513bd315\") " Nov 03 14:30:54.644010 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:54.643959 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c4ff8e1-33dc-4e0b-a735-9b64513bd315-kube-api-access-crwr2" pod "9c4ff8e1-33dc-4e0b-a735-9b64513bd315" (UID: "9c4ff8e1-33dc-4e0b-a735-9b64513bd315"). InnerVolumeSpecName "kube-api-access-crwr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:30:54.725861 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:54.722645 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crwr2\" (UniqueName: \"kubernetes.io/projected/9c4ff8e1-33dc-4e0b-a735-9b64513bd315-kube-api-access-crwr2\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:30:54.763446 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:54.763396 5060 generic.go:356] "Generic (PLEG): container finished" podID="9c4ff8e1-33dc-4e0b-a735-9b64513bd315" containerID="fa80cf8bf96e3a5c9b6affe08fddf7f8da7c118ac820819d09281745271a14de" exitCode=137 Nov 03 14:30:54.763958 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:54.763939 5060 scope.go:122] "RemoveContainer" containerID="fa80cf8bf96e3a5c9b6affe08fddf7f8da7c118ac820819d09281745271a14de" Nov 03 14:30:54.764291 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:54.764272 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7319-1c0cee" Nov 03 14:30:54.816860 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:30:54.816800 5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7319-1c0cee\" is forbidden: User \"system:node:ip-10-0-143-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-171.ec2.internal' and this object" podUID="9c4ff8e1-33dc-4e0b-a735-9b64513bd315" pod="opendatahub/test-curl-7319-1c0cee" Nov 03 14:30:54.849089 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:54.849057 5060 scope.go:122] "RemoveContainer" containerID="fa80cf8bf96e3a5c9b6affe08fddf7f8da7c118ac820819d09281745271a14de" Nov 03 14:30:54.864114 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:30:54.864065 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa80cf8bf96e3a5c9b6affe08fddf7f8da7c118ac820819d09281745271a14de\": container with ID starting with fa80cf8bf96e3a5c9b6affe08fddf7f8da7c118ac820819d09281745271a14de not found: ID does not exist" containerID="fa80cf8bf96e3a5c9b6affe08fddf7f8da7c118ac820819d09281745271a14de" Nov 03 14:30:54.864405 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:54.864369 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa80cf8bf96e3a5c9b6affe08fddf7f8da7c118ac820819d09281745271a14de"} err="failed to get container status \"fa80cf8bf96e3a5c9b6affe08fddf7f8da7c118ac820819d09281745271a14de\": rpc error: code = NotFound desc = could not find container \"fa80cf8bf96e3a5c9b6affe08fddf7f8da7c118ac820819d09281745271a14de\": container with ID starting with fa80cf8bf96e3a5c9b6affe08fddf7f8da7c118ac820819d09281745271a14de not found: ID does not exist" Nov 03 14:30:56.250957 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:30:56.250921 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c4ff8e1-33dc-4e0b-a735-9b64513bd315" path="/var/lib/kubelet/pods/9c4ff8e1-33dc-4e0b-a735-9b64513bd315/volumes" Nov 03 14:30:56.259945 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:30:56.259912 5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7319-1c0cee\" is forbidden: User \"system:node:ip-10-0-143-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-171.ec2.internal' and this object" podUID="9c4ff8e1-33dc-4e0b-a735-9b64513bd315" pod="opendatahub/test-curl-7319-1c0cee" Nov 03 14:31:01.268762 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:01.268725 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-054f11b4-r459z"] Nov 03 14:31:01.269265 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:01.269244 5060 cpu_manager.go:414] "containerMap: removing container" podUID="85bc4da4-ce30-4ea6-8a43-46f361b1f611" containerName="istio-proxy" Nov 03 14:31:01.269265 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:01.269258 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85bc4da4-ce30-4ea6-8a43-46f361b1f611" containerName="istio-proxy" Nov 03 14:31:01.269437 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:01.269277 5060 cpu_manager.go:414] "containerMap: removing container" podUID="9c4ff8e1-33dc-4e0b-a735-9b64513bd315" containerName="test-curl-7319-1c0cee" Nov 03 14:31:01.269437 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:01.269283 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c4ff8e1-33dc-4e0b-a735-9b64513bd315" containerName="test-curl-7319-1c0cee" Nov 03 14:31:01.269437 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:01.269295 5060 cpu_manager.go:414] "containerMap: removing container" podUID="85bc4da4-ce30-4ea6-8a43-46f361b1f611" containerName="istio-proxy" Nov 03 14:31:01.269437 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:01.269300 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85bc4da4-ce30-4ea6-8a43-46f361b1f611" containerName="istio-proxy" Nov 03 14:31:01.269437 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:01.269328 5060 cpu_manager.go:414] "containerMap: removing container" podUID="dea710fa-d5e0-463e-adff-8d708346f953" containerName="revoke-keys" Nov 03 14:31:01.269437 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:01.269334 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dea710fa-d5e0-463e-adff-8d708346f953" containerName="revoke-keys" Nov 03 14:31:01.269437 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:01.269431 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c4ff8e1-33dc-4e0b-a735-9b64513bd315" containerName="test-curl-7319-1c0cee" Nov 03 14:31:01.269659 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:01.269441 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="85bc4da4-ce30-4ea6-8a43-46f361b1f611" containerName="istio-proxy" Nov 03 14:31:01.269659 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:01.269448 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="dea710fa-d5e0-463e-adff-8d708346f953" containerName="revoke-keys" Nov 03 14:31:01.269659 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:01.269459 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="85bc4da4-ce30-4ea6-8a43-46f361b1f611" containerName="istio-proxy" Nov 03 14:31:01.274194 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:01.274172 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-054f11b4-r459z" Nov 03 14:31:01.295770 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:01.295739 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-054f11b4-r459z"] Nov 03 14:31:01.313895 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:01.304147 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:31:01.313895 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:01.304305 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:31:01.313895 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:01.304334 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:31:01.313895 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:01.304345 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\"" Nov 03 14:31:01.431824 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:01.431763 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzsfz\" (UniqueName: \"kubernetes.io/projected/f08e371c-e230-4bcf-9e3d-114e6c81c7d2-kube-api-access-vzsfz\") pod \"maas-api-revoke-sub-054f11b4-r459z\" (UID: \"f08e371c-e230-4bcf-9e3d-114e6c81c7d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-054f11b4-r459z" Nov 03 14:31:01.432031 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:01.431973 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f08e371c-e230-4bcf-9e3d-114e6c81c7d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-054f11b4-r459z\" (UID: \"f08e371c-e230-4bcf-9e3d-114e6c81c7d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-054f11b4-r459z" Nov 03 14:31:01.432094 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:01.432071 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f08e371c-e230-4bcf-9e3d-114e6c81c7d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-054f11b4-r459z\" (UID: \"f08e371c-e230-4bcf-9e3d-114e6c81c7d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-054f11b4-r459z" Nov 03 14:31:01.533295 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:01.533203 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f08e371c-e230-4bcf-9e3d-114e6c81c7d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-054f11b4-r459z\" (UID: \"f08e371c-e230-4bcf-9e3d-114e6c81c7d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-054f11b4-r459z" Nov 03 14:31:01.533607 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:01.533579 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f08e371c-e230-4bcf-9e3d-114e6c81c7d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-054f11b4-r459z\" (UID: \"f08e371c-e230-4bcf-9e3d-114e6c81c7d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-054f11b4-r459z" Nov 03 14:31:01.533740 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:01.533722 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzsfz\" (UniqueName: \"kubernetes.io/projected/f08e371c-e230-4bcf-9e3d-114e6c81c7d2-kube-api-access-vzsfz\") pod \"maas-api-revoke-sub-054f11b4-r459z\" (UID: \"f08e371c-e230-4bcf-9e3d-114e6c81c7d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-054f11b4-r459z" Nov 03 14:31:01.534454 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:01.534427 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f08e371c-e230-4bcf-9e3d-114e6c81c7d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-054f11b4-r459z\" (UID: \"f08e371c-e230-4bcf-9e3d-114e6c81c7d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-054f11b4-r459z" Nov 03 14:31:01.536076 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:01.536054 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f08e371c-e230-4bcf-9e3d-114e6c81c7d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-054f11b4-r459z\" (UID: \"f08e371c-e230-4bcf-9e3d-114e6c81c7d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-054f11b4-r459z" Nov 03 14:31:01.556958 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:01.556930 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzsfz\" (UniqueName: \"kubernetes.io/projected/f08e371c-e230-4bcf-9e3d-114e6c81c7d2-kube-api-access-vzsfz\") pod \"maas-api-revoke-sub-054f11b4-r459z\" (UID: \"f08e371c-e230-4bcf-9e3d-114e6c81c7d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-054f11b4-r459z" Nov 03 14:31:01.588734 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:01.588709 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-054f11b4-r459z" Nov 03 14:31:01.906621 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:01.906574 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-054f11b4-r459z"] Nov 03 14:31:01.908112 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:31:01.908084 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf08e371c_e230_4bcf_9e3d_114e6c81c7d2.slice/crio-195d2ccd9a026685449396c70a31135c01556cb70e6027cae2ddf2de699add0a WatchSource:0}: Error finding container 195d2ccd9a026685449396c70a31135c01556cb70e6027cae2ddf2de699add0a: Status 404 returned error can't find the container with id 195d2ccd9a026685449396c70a31135c01556cb70e6027cae2ddf2de699add0a Nov 03 14:31:02.849047 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:02.849001 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-054f11b4-r459z" event={"ID":"f08e371c-e230-4bcf-9e3d-114e6c81c7d2","Type":"ContainerStarted","Data":"e93523379f2673a0a82ee63f0e98a9eb05421f8a51f76748b13b7eab52582b0a"} Nov 03 14:31:02.849789 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:02.849756 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-054f11b4-r459z" event={"ID":"f08e371c-e230-4bcf-9e3d-114e6c81c7d2","Type":"ContainerStarted","Data":"195d2ccd9a026685449396c70a31135c01556cb70e6027cae2ddf2de699add0a"} Nov 03 14:31:03.859769 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:03.859729 5060 generic.go:356] "Generic (PLEG): container finished" podID="f08e371c-e230-4bcf-9e3d-114e6c81c7d2" containerID="e93523379f2673a0a82ee63f0e98a9eb05421f8a51f76748b13b7eab52582b0a" exitCode=0 Nov 03 14:31:03.860218 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:03.859810 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-054f11b4-r459z" event={"ID":"f08e371c-e230-4bcf-9e3d-114e6c81c7d2","Type":"ContainerDied","Data":"e93523379f2673a0a82ee63f0e98a9eb05421f8a51f76748b13b7eab52582b0a"} Nov 03 14:31:05.017642 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:05.017611 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-054f11b4-r459z" Nov 03 14:31:05.179800 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:05.179718 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f08e371c-e230-4bcf-9e3d-114e6c81c7d2-kube-api-access-vzsfz\" (UniqueName: \"kubernetes.io/projected/f08e371c-e230-4bcf-9e3d-114e6c81c7d2-kube-api-access-vzsfz\") pod \"f08e371c-e230-4bcf-9e3d-114e6c81c7d2\" (UID: \"f08e371c-e230-4bcf-9e3d-114e6c81c7d2\") " Nov 03 14:31:05.179960 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:05.179847 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f08e371c-e230-4bcf-9e3d-114e6c81c7d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f08e371c-e230-4bcf-9e3d-114e6c81c7d2-maas-api-openshift-ca\") pod \"f08e371c-e230-4bcf-9e3d-114e6c81c7d2\" (UID: \"f08e371c-e230-4bcf-9e3d-114e6c81c7d2\") " Nov 03 14:31:05.179960 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:05.179878 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f08e371c-e230-4bcf-9e3d-114e6c81c7d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f08e371c-e230-4bcf-9e3d-114e6c81c7d2-maas-api-xks-ca\") pod \"f08e371c-e230-4bcf-9e3d-114e6c81c7d2\" (UID: \"f08e371c-e230-4bcf-9e3d-114e6c81c7d2\") " Nov 03 14:31:05.180282 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:05.180243 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f08e371c-e230-4bcf-9e3d-114e6c81c7d2-maas-api-openshift-ca" pod "f08e371c-e230-4bcf-9e3d-114e6c81c7d2" (UID: "f08e371c-e230-4bcf-9e3d-114e6c81c7d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:31:05.181852 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:05.181813 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f08e371c-e230-4bcf-9e3d-114e6c81c7d2-maas-api-xks-ca" pod "f08e371c-e230-4bcf-9e3d-114e6c81c7d2" (UID: "f08e371c-e230-4bcf-9e3d-114e6c81c7d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:31:05.181974 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:05.181906 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f08e371c-e230-4bcf-9e3d-114e6c81c7d2-kube-api-access-vzsfz" pod "f08e371c-e230-4bcf-9e3d-114e6c81c7d2" (UID: "f08e371c-e230-4bcf-9e3d-114e6c81c7d2"). InnerVolumeSpecName "kube-api-access-vzsfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:31:05.281539 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:05.281501 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzsfz\" (UniqueName: \"kubernetes.io/projected/f08e371c-e230-4bcf-9e3d-114e6c81c7d2-kube-api-access-vzsfz\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:31:05.281539 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:05.281533 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f08e371c-e230-4bcf-9e3d-114e6c81c7d2-maas-api-openshift-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:31:05.281539 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:05.281542 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f08e371c-e230-4bcf-9e3d-114e6c81c7d2-maas-api-xks-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:31:05.878569 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:05.878526 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-054f11b4-r459z" event={"ID":"f08e371c-e230-4bcf-9e3d-114e6c81c7d2","Type":"ContainerDied","Data":"195d2ccd9a026685449396c70a31135c01556cb70e6027cae2ddf2de699add0a"} Nov 03 14:31:05.878793 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:05.878592 5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="195d2ccd9a026685449396c70a31135c01556cb70e6027cae2ddf2de699add0a" Nov 03 14:31:05.878793 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:05.878657 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-054f11b4-r459z" Nov 03 14:31:06.736860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:06.693666 5060 scope.go:122] "RemoveContainer" containerID="0d62004ea2b310d6b52c8361502c989dab4034e3a0aa26f347dc4ca8bf46238f" Nov 03 14:31:06.951110 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:06.933097 5060 scope.go:122] "RemoveContainer" containerID="e3987d59782d17e0e42aa24b981c6a532507b921df99ff07a73cb159792c907d" Nov 03 14:31:07.038860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:07.016672 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/3.log" Nov 03 14:31:07.038860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:07.035643 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/3.log" Nov 03 14:31:07.083108 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:07.083074 5060 scope.go:122] "RemoveContainer" containerID="a970c8bdc1982d4e9df5fce8a1a610dbc13a8da263f5d9412d01d1b24908e023" Nov 03 14:31:09.441497 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:31:09.441455 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:31:15.802756 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:15.802719 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf"] Nov 03 14:31:15.803574 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:15.803549 5060 cpu_manager.go:414] "containerMap: removing container" podUID="f08e371c-e230-4bcf-9e3d-114e6c81c7d2" containerName="revoke-keys" Nov 03 14:31:15.803574 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:15.803571 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f08e371c-e230-4bcf-9e3d-114e6c81c7d2" containerName="revoke-keys" Nov 03 14:31:15.803779 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:15.803694 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f08e371c-e230-4bcf-9e3d-114e6c81c7d2" containerName="revoke-keys" Nov 03 14:31:15.808908 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:15.808887 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" Nov 03 14:31:15.827051 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:15.827017 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae45680-07cf-4414-851a-2272d68ccc21-workload-certs\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf\" (UID: \"9ae45680-07cf-4414-851a-2272d68ccc21\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" Nov 03 14:31:15.827168 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:15.827148 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae45680-07cf-4414-851a-2272d68ccc21-istiod-ca-cert\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf\" (UID: \"9ae45680-07cf-4414-851a-2272d68ccc21\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" Nov 03 14:31:15.827225 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:15.827214 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae45680-07cf-4414-851a-2272d68ccc21-istio-podinfo\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf\" (UID: \"9ae45680-07cf-4414-851a-2272d68ccc21\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" Nov 03 14:31:15.827270 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:15.827250 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae45680-07cf-4414-851a-2272d68ccc21-istio-envoy\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf\" (UID: \"9ae45680-07cf-4414-851a-2272d68ccc21\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" Nov 03 14:31:15.827322 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:15.827310 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae45680-07cf-4414-851a-2272d68ccc21-workload-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf\" (UID: \"9ae45680-07cf-4414-851a-2272d68ccc21\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" Nov 03 14:31:15.827358 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:15.827349 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d4ll\" (UniqueName: \"kubernetes.io/projected/9ae45680-07cf-4414-851a-2272d68ccc21-kube-api-access-7d4ll\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf\" (UID: \"9ae45680-07cf-4414-851a-2272d68ccc21\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" Nov 03 14:31:15.827423 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:15.827409 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae45680-07cf-4414-851a-2272d68ccc21-credential-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf\" (UID: \"9ae45680-07cf-4414-851a-2272d68ccc21\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" Nov 03 14:31:15.827460 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:15.827438 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae45680-07cf-4414-851a-2272d68ccc21-istio-data\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf\" (UID: \"9ae45680-07cf-4414-851a-2272d68ccc21\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" Nov 03 14:31:15.827562 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:15.827541 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ae45680-07cf-4414-851a-2272d68ccc21-istio-token\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf\" (UID: \"9ae45680-07cf-4414-851a-2272d68ccc21\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" Nov 03 14:31:15.847850 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:15.847785 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf"] Nov 03 14:31:15.928211 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:15.928173 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae45680-07cf-4414-851a-2272d68ccc21-workload-certs\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf\" (UID: \"9ae45680-07cf-4414-851a-2272d68ccc21\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" Nov 03 14:31:15.928413 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:15.928277 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae45680-07cf-4414-851a-2272d68ccc21-istiod-ca-cert\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf\" (UID: \"9ae45680-07cf-4414-851a-2272d68ccc21\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" Nov 03 14:31:15.928413 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:15.928354 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae45680-07cf-4414-851a-2272d68ccc21-istio-podinfo\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf\" (UID: \"9ae45680-07cf-4414-851a-2272d68ccc21\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" Nov 03 14:31:15.928413 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:15.928393 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae45680-07cf-4414-851a-2272d68ccc21-istio-envoy\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf\" (UID: \"9ae45680-07cf-4414-851a-2272d68ccc21\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" Nov 03 14:31:15.928591 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:15.928422 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae45680-07cf-4414-851a-2272d68ccc21-workload-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf\" (UID: \"9ae45680-07cf-4414-851a-2272d68ccc21\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" Nov 03 14:31:15.928591 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:15.928476 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d4ll\" (UniqueName: \"kubernetes.io/projected/9ae45680-07cf-4414-851a-2272d68ccc21-kube-api-access-7d4ll\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf\" (UID: \"9ae45680-07cf-4414-851a-2272d68ccc21\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" Nov 03 14:31:15.928591 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:15.928517 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae45680-07cf-4414-851a-2272d68ccc21-credential-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf\" (UID: \"9ae45680-07cf-4414-851a-2272d68ccc21\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" Nov 03 14:31:15.928591 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:15.928548 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae45680-07cf-4414-851a-2272d68ccc21-istio-data\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf\" (UID: \"9ae45680-07cf-4414-851a-2272d68ccc21\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" Nov 03 14:31:15.928823 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:15.928609 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ae45680-07cf-4414-851a-2272d68ccc21-istio-token\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf\" (UID: \"9ae45680-07cf-4414-851a-2272d68ccc21\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" Nov 03 14:31:15.928823 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:15.928657 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae45680-07cf-4414-851a-2272d68ccc21-workload-certs\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf\" (UID: \"9ae45680-07cf-4414-851a-2272d68ccc21\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" Nov 03 14:31:15.928986 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:15.928947 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae45680-07cf-4414-851a-2272d68ccc21-credential-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf\" (UID: \"9ae45680-07cf-4414-851a-2272d68ccc21\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" Nov 03 14:31:15.929107 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:15.928964 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae45680-07cf-4414-851a-2272d68ccc21-istio-data\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf\" (UID: \"9ae45680-07cf-4414-851a-2272d68ccc21\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" Nov 03 14:31:15.929167 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:15.929130 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae45680-07cf-4414-851a-2272d68ccc21-istiod-ca-cert\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf\" (UID: \"9ae45680-07cf-4414-851a-2272d68ccc21\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" Nov 03 14:31:15.929218 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:15.929204 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae45680-07cf-4414-851a-2272d68ccc21-workload-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf\" (UID: \"9ae45680-07cf-4414-851a-2272d68ccc21\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" Nov 03 14:31:15.931074 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:15.931016 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae45680-07cf-4414-851a-2272d68ccc21-istio-envoy\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf\" (UID: \"9ae45680-07cf-4414-851a-2272d68ccc21\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" Nov 03 14:31:15.931481 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:15.931458 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae45680-07cf-4414-851a-2272d68ccc21-istio-podinfo\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf\" (UID: \"9ae45680-07cf-4414-851a-2272d68ccc21\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" Nov 03 14:31:15.951108 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:15.951078 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ae45680-07cf-4414-851a-2272d68ccc21-istio-token\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf\" (UID: \"9ae45680-07cf-4414-851a-2272d68ccc21\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" Nov 03 14:31:15.951237 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:15.951157 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d4ll\" (UniqueName: \"kubernetes.io/projected/9ae45680-07cf-4414-851a-2272d68ccc21-kube-api-access-7d4ll\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf\" (UID: \"9ae45680-07cf-4414-851a-2272d68ccc21\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" Nov 03 14:31:16.123011 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:16.122924 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" Nov 03 14:31:16.450566 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:16.450537 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf"] Nov 03 14:31:16.451958 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:31:16.451937 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ae45680_07cf_4414_851a_2272d68ccc21.slice/crio-0bdaafa2927aeadec2e9468abe7d9a5e2b7c35a4eccd9a5db093c730c04234bf WatchSource:0}: Error finding container 0bdaafa2927aeadec2e9468abe7d9a5e2b7c35a4eccd9a5db093c730c04234bf: Status 404 returned error can't find the container with id 0bdaafa2927aeadec2e9468abe7d9a5e2b7c35a4eccd9a5db093c730c04234bf Nov 03 14:31:16.454864 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:16.454803 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:31:16.454951 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:16.454907 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:31:16.454989 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:16.454953 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:31:17.016166 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:17.016124 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" event={"ID":"9ae45680-07cf-4414-851a-2272d68ccc21","Type":"ContainerStarted","Data":"6fc17d31664a5cac91b13b36795cac97370c2a816cf5cf94f81ac1fbafd2f6a2"} Nov 03 14:31:17.016579 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:17.016192 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" event={"ID":"9ae45680-07cf-4414-851a-2272d68ccc21","Type":"ContainerStarted","Data":"0bdaafa2927aeadec2e9468abe7d9a5e2b7c35a4eccd9a5db093c730c04234bf"} Nov 03 14:31:17.090204 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:17.090156 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" podStartSLOduration=2.090144 podStartE2EDuration="2.090144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:31:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:31:17.075708923 +0000 UTC m=+1210.964498504" watchObservedRunningTime="2026-11-03 14:31:17.090144 +0000 UTC m=+1210.978933581" Nov 03 14:31:17.123977 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:17.123941 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" Nov 03 14:31:17.125592 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:17.125566 5060 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 14:31:17.125701 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:17.125613 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" podUID="9ae45680-07cf-4414-851a-2272d68ccc21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 14:31:17.731857 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:17.721483 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz"] Nov 03 14:31:17.738203 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:17.738176 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" Nov 03 14:31:17.738753 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:17.738723 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz"] Nov 03 14:31:17.755654 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:17.755619 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50995ad2-461f-44c5-97e9-8e4f1922e39d-istio-token\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz\" (UID: \"50995ad2-461f-44c5-97e9-8e4f1922e39d\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" Nov 03 14:31:17.755816 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:17.755719 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50995ad2-461f-44c5-97e9-8e4f1922e39d-istio-data\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz\" (UID: \"50995ad2-461f-44c5-97e9-8e4f1922e39d\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" Nov 03 14:31:17.755964 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:17.755823 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50995ad2-461f-44c5-97e9-8e4f1922e39d-istiod-ca-cert\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz\" (UID: \"50995ad2-461f-44c5-97e9-8e4f1922e39d\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" Nov 03 14:31:17.756029 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:17.755953 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50995ad2-461f-44c5-97e9-8e4f1922e39d-workload-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz\" (UID: \"50995ad2-461f-44c5-97e9-8e4f1922e39d\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" Nov 03 14:31:17.756029 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:17.756013 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50995ad2-461f-44c5-97e9-8e4f1922e39d-workload-certs\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz\" (UID: \"50995ad2-461f-44c5-97e9-8e4f1922e39d\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" Nov 03 14:31:17.756141 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:17.756127 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50995ad2-461f-44c5-97e9-8e4f1922e39d-istio-envoy\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz\" (UID: \"50995ad2-461f-44c5-97e9-8e4f1922e39d\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" Nov 03 14:31:17.756197 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:17.756183 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50995ad2-461f-44c5-97e9-8e4f1922e39d-credential-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz\" (UID: \"50995ad2-461f-44c5-97e9-8e4f1922e39d\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" Nov 03 14:31:17.756252 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:17.756222 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50995ad2-461f-44c5-97e9-8e4f1922e39d-istio-podinfo\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz\" (UID: \"50995ad2-461f-44c5-97e9-8e4f1922e39d\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" Nov 03 14:31:17.756317 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:17.756250 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rznzp\" (UniqueName: \"kubernetes.io/projected/50995ad2-461f-44c5-97e9-8e4f1922e39d-kube-api-access-rznzp\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz\" (UID: \"50995ad2-461f-44c5-97e9-8e4f1922e39d\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" Nov 03 14:31:17.857225 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:17.857168 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50995ad2-461f-44c5-97e9-8e4f1922e39d-istio-envoy\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz\" (UID: \"50995ad2-461f-44c5-97e9-8e4f1922e39d\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" Nov 03 14:31:17.857225 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:17.857239 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50995ad2-461f-44c5-97e9-8e4f1922e39d-credential-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz\" (UID: \"50995ad2-461f-44c5-97e9-8e4f1922e39d\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" Nov 03 14:31:17.857511 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:17.857266 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50995ad2-461f-44c5-97e9-8e4f1922e39d-istio-podinfo\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz\" (UID: \"50995ad2-461f-44c5-97e9-8e4f1922e39d\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" Nov 03 14:31:17.857511 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:17.857286 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rznzp\" (UniqueName: \"kubernetes.io/projected/50995ad2-461f-44c5-97e9-8e4f1922e39d-kube-api-access-rznzp\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz\" (UID: \"50995ad2-461f-44c5-97e9-8e4f1922e39d\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" Nov 03 14:31:17.857511 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:17.857327 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50995ad2-461f-44c5-97e9-8e4f1922e39d-istio-token\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz\" (UID: \"50995ad2-461f-44c5-97e9-8e4f1922e39d\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" Nov 03 14:31:17.857511 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:17.857361 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50995ad2-461f-44c5-97e9-8e4f1922e39d-istio-data\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz\" (UID: \"50995ad2-461f-44c5-97e9-8e4f1922e39d\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" Nov 03 14:31:17.857511 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:17.857416 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50995ad2-461f-44c5-97e9-8e4f1922e39d-istiod-ca-cert\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz\" (UID: \"50995ad2-461f-44c5-97e9-8e4f1922e39d\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" Nov 03 14:31:17.857511 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:17.857480 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50995ad2-461f-44c5-97e9-8e4f1922e39d-workload-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz\" (UID: \"50995ad2-461f-44c5-97e9-8e4f1922e39d\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" Nov 03 14:31:17.857511 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:17.857508 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50995ad2-461f-44c5-97e9-8e4f1922e39d-workload-certs\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz\" (UID: \"50995ad2-461f-44c5-97e9-8e4f1922e39d\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" Nov 03 14:31:17.857936 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:17.857586 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50995ad2-461f-44c5-97e9-8e4f1922e39d-credential-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz\" (UID: \"50995ad2-461f-44c5-97e9-8e4f1922e39d\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" Nov 03 14:31:17.857936 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:17.857816 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50995ad2-461f-44c5-97e9-8e4f1922e39d-istio-data\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz\" (UID: \"50995ad2-461f-44c5-97e9-8e4f1922e39d\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" Nov 03 14:31:17.857936 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:17.857918 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50995ad2-461f-44c5-97e9-8e4f1922e39d-workload-certs\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz\" (UID: \"50995ad2-461f-44c5-97e9-8e4f1922e39d\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" Nov 03 14:31:17.858100 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:17.858082 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50995ad2-461f-44c5-97e9-8e4f1922e39d-workload-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz\" (UID: \"50995ad2-461f-44c5-97e9-8e4f1922e39d\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" Nov 03 14:31:17.858253 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:17.858231 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50995ad2-461f-44c5-97e9-8e4f1922e39d-istiod-ca-cert\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz\" (UID: \"50995ad2-461f-44c5-97e9-8e4f1922e39d\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" Nov 03 14:31:17.859646 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:17.859625 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50995ad2-461f-44c5-97e9-8e4f1922e39d-istio-envoy\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz\" (UID: \"50995ad2-461f-44c5-97e9-8e4f1922e39d\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" Nov 03 14:31:17.859870 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:17.859851 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50995ad2-461f-44c5-97e9-8e4f1922e39d-istio-podinfo\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz\" (UID: \"50995ad2-461f-44c5-97e9-8e4f1922e39d\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" Nov 03 14:31:17.879186 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:17.879155 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50995ad2-461f-44c5-97e9-8e4f1922e39d-istio-token\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz\" (UID: \"50995ad2-461f-44c5-97e9-8e4f1922e39d\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" Nov 03 14:31:17.879311 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:17.879286 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rznzp\" (UniqueName: \"kubernetes.io/projected/50995ad2-461f-44c5-97e9-8e4f1922e39d-kube-api-access-rznzp\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz\" (UID: \"50995ad2-461f-44c5-97e9-8e4f1922e39d\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" Nov 03 14:31:18.064884 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:18.064847 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" Nov 03 14:31:18.131717 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:18.131678 5060 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 14:31:18.131940 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:18.131745 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" podUID="9ae45680-07cf-4414-851a-2272d68ccc21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 14:31:18.435857 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:18.434030 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz"] Nov 03 14:31:18.437776 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:31:18.437741 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50995ad2_461f_44c5_97e9_8e4f1922e39d.slice/crio-48db54768d0e4a64fcf3f677ab6276dae1d21b57839c7e20e96ee4abdcdbb2eb WatchSource:0}: Error finding container 48db54768d0e4a64fcf3f677ab6276dae1d21b57839c7e20e96ee4abdcdbb2eb: Status 404 returned error can't find the container with id 48db54768d0e4a64fcf3f677ab6276dae1d21b57839c7e20e96ee4abdcdbb2eb Nov 03 14:31:18.441363 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:18.441322 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:31:18.441470 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:18.441407 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:31:18.441470 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:18.441447 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:31:19.043479 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:19.043436 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" event={"ID":"50995ad2-461f-44c5-97e9-8e4f1922e39d","Type":"ContainerStarted","Data":"34cc929db15530dee75908267588804436663ee93d42c1d191f292ae9a1705cc"} Nov 03 14:31:19.043745 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:19.043543 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" event={"ID":"50995ad2-461f-44c5-97e9-8e4f1922e39d","Type":"ContainerStarted","Data":"48db54768d0e4a64fcf3f677ab6276dae1d21b57839c7e20e96ee4abdcdbb2eb"} Nov 03 14:31:19.065684 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:19.065655 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" Nov 03 14:31:19.066733 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:19.066710 5060 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 14:31:19.066830 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:19.066762 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" podUID="50995ad2-461f-44c5-97e9-8e4f1922e39d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 14:31:19.128857 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:19.128301 5060 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 14:31:19.128857 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:19.128363 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" podUID="9ae45680-07cf-4414-851a-2272d68ccc21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 14:31:19.131585 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:19.131525 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" podStartSLOduration=2.131507391 podStartE2EDuration="2.131507391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:31:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:31:19.128078281 +0000 UTC m=+1213.016867895" watchObservedRunningTime="2026-11-03 14:31:19.131507391 +0000 UTC m=+1213.020297005" Nov 03 14:31:20.065516 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:20.065476 5060 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 14:31:20.065695 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:20.065534 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" podUID="50995ad2-461f-44c5-97e9-8e4f1922e39d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 14:31:20.129255 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:20.129216 5060 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 14:31:20.129825 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:20.129795 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" podUID="9ae45680-07cf-4414-851a-2272d68ccc21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 14:31:21.068857 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:21.068558 5060 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 14:31:21.068857 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:21.068632 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" podUID="50995ad2-461f-44c5-97e9-8e4f1922e39d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 14:31:21.123527 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:21.123487 5060 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 14:31:21.124163 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:21.124124 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" podUID="9ae45680-07cf-4414-851a-2272d68ccc21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 14:31:22.072449 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:22.072415 5060 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 14:31:22.073083 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:22.073052 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" podUID="50995ad2-461f-44c5-97e9-8e4f1922e39d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 14:31:22.131863 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:22.130348 5060 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 14:31:22.131863 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:22.130409 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" podUID="9ae45680-07cf-4414-851a-2272d68ccc21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 14:31:22.921472 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:22.921433 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/distinct-2-w3-156ae1-kserve-5988894d76-vsdlm" podUID="59dcdef3-f88e-4463-a0dd-1917e915a516" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 14:31:23.066596 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:23.066183 5060 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 14:31:23.066596 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:23.066240 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" podUID="50995ad2-461f-44c5-97e9-8e4f1922e39d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 14:31:23.126882 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:23.125448 5060 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 14:31:23.126882 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:23.125505 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" podUID="9ae45680-07cf-4414-851a-2272d68ccc21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 14:31:23.418929 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:23.404236 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk" podUID="49b53157-30cd-4fe2-8cca-a46e7dbec6f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Nov 03 14:31:23.444860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:23.441715 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98" podUID="3aaaeea4-134f-4d17-a180-b4857c6425d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 14:31:23.657857 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:23.655818 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-2d0da2/sub-test-model-2d0da2-kserve-6f7f89db79-kffdh" podUID="3dc0b682-756a-4736-9d3a-ab20195b4bcb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 14:31:23.766110 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:23.766058 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-4a67e25f/e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p" podUID="0476cf91-73a3-4392-a1c7-1dc62e2a5b9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 14:31:24.072376 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:24.072278 5060 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 14:31:24.072660 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:24.072627 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" podUID="50995ad2-461f-44c5-97e9-8e4f1922e39d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 14:31:24.127345 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:24.127307 5060 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 14:31:24.127998 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:24.127948 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" podUID="9ae45680-07cf-4414-851a-2272d68ccc21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 14:31:25.065587 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:25.065556 5060 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 14:31:25.065761 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:25.065609 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" podUID="50995ad2-461f-44c5-97e9-8e4f1922e39d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 14:31:25.128622 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:25.128588 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" Nov 03 14:31:25.129071 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:25.129050 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" Nov 03 14:31:25.129793 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:25.129774 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" Nov 03 14:31:26.066037 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:26.065995 5060 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 14:31:26.066256 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:26.066048 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" podUID="50995ad2-461f-44c5-97e9-8e4f1922e39d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 14:31:27.070282 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:27.070252 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" Nov 03 14:31:27.070763 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:27.070676 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" Nov 03 14:31:27.071632 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:27.071607 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" Nov 03 14:31:29.040151 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:29.040111 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55be1c47-tqqbg"] Nov 03 14:31:29.052457 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:29.052430 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55be1c47-tqqbg"] Nov 03 14:31:29.489191 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:29.489108 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1daa315-85f9b"] Nov 03 14:31:29.493292 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:29.493268 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1daa315-85f9b" Nov 03 14:31:29.516608 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:29.516580 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1daa315-85f9b"] Nov 03 14:31:29.523987 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:29.523965 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:31:29.524290 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:29.524270 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:31:29.524401 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:29.524350 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\"" Nov 03 14:31:29.524401 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:29.524389 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:31:29.663648 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:29.663612 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/881ff6f4-04c1-471e-8ade-1c61fe6486a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c1daa315-85f9b\" (UID: \"881ff6f4-04c1-471e-8ade-1c61fe6486a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1daa315-85f9b" Nov 03 14:31:29.663828 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:29.663758 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/881ff6f4-04c1-471e-8ade-1c61fe6486a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c1daa315-85f9b\" (UID: \"881ff6f4-04c1-471e-8ade-1c61fe6486a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1daa315-85f9b" Nov 03 14:31:29.663932 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:29.663916 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5p4z\" (UniqueName: \"kubernetes.io/projected/881ff6f4-04c1-471e-8ade-1c61fe6486a4-kube-api-access-q5p4z\") pod \"maas-api-revoke-keys-e2e-ait-c1daa315-85f9b\" (UID: \"881ff6f4-04c1-471e-8ade-1c61fe6486a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1daa315-85f9b" Nov 03 14:31:29.765447 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:29.765408 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/881ff6f4-04c1-471e-8ade-1c61fe6486a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c1daa315-85f9b\" (UID: \"881ff6f4-04c1-471e-8ade-1c61fe6486a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1daa315-85f9b" Nov 03 14:31:29.765620 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:29.765552 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5p4z\" (UniqueName: \"kubernetes.io/projected/881ff6f4-04c1-471e-8ade-1c61fe6486a4-kube-api-access-q5p4z\") pod \"maas-api-revoke-keys-e2e-ait-c1daa315-85f9b\" (UID: \"881ff6f4-04c1-471e-8ade-1c61fe6486a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1daa315-85f9b" Nov 03 14:31:29.765706 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:29.765629 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/881ff6f4-04c1-471e-8ade-1c61fe6486a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c1daa315-85f9b\" (UID: \"881ff6f4-04c1-471e-8ade-1c61fe6486a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1daa315-85f9b" Nov 03 14:31:29.766105 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:29.766080 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/881ff6f4-04c1-471e-8ade-1c61fe6486a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c1daa315-85f9b\" (UID: \"881ff6f4-04c1-471e-8ade-1c61fe6486a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1daa315-85f9b" Nov 03 14:31:29.768082 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:29.768053 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/881ff6f4-04c1-471e-8ade-1c61fe6486a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c1daa315-85f9b\" (UID: \"881ff6f4-04c1-471e-8ade-1c61fe6486a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1daa315-85f9b" Nov 03 14:31:29.788156 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:29.788125 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5p4z\" (UniqueName: \"kubernetes.io/projected/881ff6f4-04c1-471e-8ade-1c61fe6486a4-kube-api-access-q5p4z\") pod \"maas-api-revoke-keys-e2e-ait-c1daa315-85f9b\" (UID: \"881ff6f4-04c1-471e-8ade-1c61fe6486a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1daa315-85f9b" Nov 03 14:31:29.807889 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:29.807853 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1daa315-85f9b" Nov 03 14:31:30.120045 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:30.120022 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1daa315-85f9b"] Nov 03 14:31:30.122045 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:31:30.122012 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod881ff6f4_04c1_471e_8ade_1c61fe6486a4.slice/crio-4f5b9b7a746851235193c00af9797a5789412618e4267f591a84d3ac4988fae1 WatchSource:0}: Error finding container 4f5b9b7a746851235193c00af9797a5789412618e4267f591a84d3ac4988fae1: Status 404 returned error can't find the container with id 4f5b9b7a746851235193c00af9797a5789412618e4267f591a84d3ac4988fae1 Nov 03 14:31:30.241104 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:30.241059 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1daa315-85f9b" event={"ID":"881ff6f4-04c1-471e-8ade-1c61fe6486a4","Type":"ContainerStarted","Data":"4f5b9b7a746851235193c00af9797a5789412618e4267f591a84d3ac4988fae1"} Nov 03 14:31:30.251645 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:30.251613 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="209a19fc-8c5d-406a-9b4d-32a4e9ff49f3" path="/var/lib/kubelet/pods/209a19fc-8c5d-406a-9b4d-32a4e9ff49f3/volumes" Nov 03 14:31:31.250154 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:31.250070 5060 generic.go:356] "Generic (PLEG): container finished" podID="881ff6f4-04c1-471e-8ade-1c61fe6486a4" containerID="8efab72153bc7b881341f7f0e7289e6e0d0ad4dd43e16bfa6669a3ca77ce78a5" exitCode=0 Nov 03 14:31:31.250572 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:31.250163 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1daa315-85f9b" event={"ID":"881ff6f4-04c1-471e-8ade-1c61fe6486a4","Type":"ContainerDied","Data":"8efab72153bc7b881341f7f0e7289e6e0d0ad4dd43e16bfa6669a3ca77ce78a5"} Nov 03 14:31:32.500178 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:32.500151 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1daa315-85f9b" Nov 03 14:31:32.609911 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:32.609571 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/881ff6f4-04c1-471e-8ade-1c61fe6486a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/881ff6f4-04c1-471e-8ade-1c61fe6486a4-maas-api-xks-ca\") pod \"881ff6f4-04c1-471e-8ade-1c61fe6486a4\" (UID: \"881ff6f4-04c1-471e-8ade-1c61fe6486a4\") " Nov 03 14:31:32.609911 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:32.609675 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/881ff6f4-04c1-471e-8ade-1c61fe6486a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/881ff6f4-04c1-471e-8ade-1c61fe6486a4-maas-api-openshift-ca\") pod \"881ff6f4-04c1-471e-8ade-1c61fe6486a4\" (UID: \"881ff6f4-04c1-471e-8ade-1c61fe6486a4\") " Nov 03 14:31:32.609911 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:32.609859 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/881ff6f4-04c1-471e-8ade-1c61fe6486a4-kube-api-access-q5p4z\" (UniqueName: \"kubernetes.io/projected/881ff6f4-04c1-471e-8ade-1c61fe6486a4-kube-api-access-q5p4z\") pod \"881ff6f4-04c1-471e-8ade-1c61fe6486a4\" (UID: \"881ff6f4-04c1-471e-8ade-1c61fe6486a4\") " Nov 03 14:31:32.614295 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:32.612871 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/881ff6f4-04c1-471e-8ade-1c61fe6486a4-maas-api-openshift-ca" pod "881ff6f4-04c1-471e-8ade-1c61fe6486a4" (UID: "881ff6f4-04c1-471e-8ade-1c61fe6486a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:31:32.647899 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:32.640376 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/881ff6f4-04c1-471e-8ade-1c61fe6486a4-maas-api-xks-ca" pod "881ff6f4-04c1-471e-8ade-1c61fe6486a4" (UID: "881ff6f4-04c1-471e-8ade-1c61fe6486a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:31:32.647899 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:32.640889 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/881ff6f4-04c1-471e-8ade-1c61fe6486a4-kube-api-access-q5p4z" pod "881ff6f4-04c1-471e-8ade-1c61fe6486a4" (UID: "881ff6f4-04c1-471e-8ade-1c61fe6486a4"). InnerVolumeSpecName "kube-api-access-q5p4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:31:32.712113 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:32.711743 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5p4z\" (UniqueName: \"kubernetes.io/projected/881ff6f4-04c1-471e-8ade-1c61fe6486a4-kube-api-access-q5p4z\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:31:32.712113 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:32.711785 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/881ff6f4-04c1-471e-8ade-1c61fe6486a4-maas-api-xks-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:31:32.712113 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:32.711802 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/881ff6f4-04c1-471e-8ade-1c61fe6486a4-maas-api-openshift-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:31:33.283856 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:33.282919 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1daa315-85f9b" event={"ID":"881ff6f4-04c1-471e-8ade-1c61fe6486a4","Type":"ContainerDied","Data":"4f5b9b7a746851235193c00af9797a5789412618e4267f591a84d3ac4988fae1"} Nov 03 14:31:33.283856 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:33.282983 5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f5b9b7a746851235193c00af9797a5789412618e4267f591a84d3ac4988fae1" Nov 03 14:31:33.283856 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:33.283047 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1daa315-85f9b" Nov 03 14:31:39.439380 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:31:39.439343 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:31:42.836334 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:42.836292 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1daa315-85f9b"] Nov 03 14:31:42.848954 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:42.848915 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1daa315-85f9b"] Nov 03 14:31:44.251038 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:44.250999 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="881ff6f4-04c1-471e-8ade-1c61fe6486a4" path="/var/lib/kubelet/pods/881ff6f4-04c1-471e-8ade-1c61fe6486a4/volumes" Nov 03 14:31:45.043122 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:45.043082 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9454069-48k8g"] Nov 03 14:31:45.059654 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:45.059612 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9454069-48k8g"] Nov 03 14:31:46.328291 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:46.328252 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23aa4bb8-5671-497f-9792-4bba9a67ed61" path="/var/lib/kubelet/pods/23aa4bb8-5671-497f-9792-4bba9a67ed61/volumes" Nov 03 14:31:48.995537 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:48.995186 5060 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:31:48.995537 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:48.995237 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" podUID="f859590f-c6cb-4b6c-bfd2-ebde80002e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:31:50.814277 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:50.814235 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg"] Nov 03 14:31:50.814770 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:50.814466 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" podUID="76bc22bf-ed6f-4f6b-a225-5611aa3b73b0" containerName="istio-proxy" containerID="cri-o://84208702fd3a29e1d27ad3e868908ab06f7b863861cf91399568acab9f972c7f" gracePeriod=30 Nov 03 14:31:50.829523 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:50.829492 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs"] Nov 03 14:31:50.830018 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:50.829964 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" podUID="44c38065-cf4a-47fd-a80f-9784767ae3ea" containerName="istio-proxy" containerID="cri-o://3b816520e9f4520b8feb02852c83b1fa3c11327e768a0a39b97bc0b347b3a533" gracePeriod=30 Nov 03 14:31:56.148388 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.148353 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" Nov 03 14:31:56.188093 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.188071 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" Nov 03 14:31:56.194272 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.194251 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44c38065-cf4a-47fd-a80f-9784767ae3ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c38065-cf4a-47fd-a80f-9784767ae3ea-workload-socket\") pod \"44c38065-cf4a-47fd-a80f-9784767ae3ea\" (UID: \"44c38065-cf4a-47fd-a80f-9784767ae3ea\") " Nov 03 14:31:56.194355 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.194334 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44c38065-cf4a-47fd-a80f-9784767ae3ea-kube-api-access-lln5r\" (UniqueName: \"kubernetes.io/projected/44c38065-cf4a-47fd-a80f-9784767ae3ea-kube-api-access-lln5r\") pod \"44c38065-cf4a-47fd-a80f-9784767ae3ea\" (UID: \"44c38065-cf4a-47fd-a80f-9784767ae3ea\") " Nov 03 14:31:56.194393 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.194361 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44c38065-cf4a-47fd-a80f-9784767ae3ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c38065-cf4a-47fd-a80f-9784767ae3ea-credential-socket\") pod \"44c38065-cf4a-47fd-a80f-9784767ae3ea\" (UID: \"44c38065-cf4a-47fd-a80f-9784767ae3ea\") " Nov 03 14:31:56.194393 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.194378 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-kube-api-access-c6fnm\" (UniqueName: \"kubernetes.io/projected/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-kube-api-access-c6fnm\") pod \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\" (UID: \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\") " Nov 03 14:31:56.194490 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.194410 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-workload-socket\") pod \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\" (UID: \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\") " Nov 03 14:31:56.194490 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.194447 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-istio-token\" (UniqueName: \"kubernetes.io/projected/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-istio-token\") pod \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\" (UID: \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\") " Nov 03 14:31:56.194490 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.194475 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c38065-cf4a-47fd-a80f-9784767ae3ea-workload-socket" pod "44c38065-cf4a-47fd-a80f-9784767ae3ea" (UID: "44c38065-cf4a-47fd-a80f-9784767ae3ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:31:56.194647 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.194492 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44c38065-cf4a-47fd-a80f-9784767ae3ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44c38065-cf4a-47fd-a80f-9784767ae3ea-istio-envoy\") pod \"44c38065-cf4a-47fd-a80f-9784767ae3ea\" (UID: \"44c38065-cf4a-47fd-a80f-9784767ae3ea\") " Nov 03 14:31:56.194647 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.194530 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-istio-envoy\") pod \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\" (UID: \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\") " Nov 03 14:31:56.194647 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.194562 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-istiod-ca-cert\") pod \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\" (UID: \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\") " Nov 03 14:31:56.194647 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.194597 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-workload-certs\") pod \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\" (UID: \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\") " Nov 03 14:31:56.194897 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.194649 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-istio-podinfo\") pod \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\" (UID: \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\") " Nov 03 14:31:56.194897 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.194683 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c38065-cf4a-47fd-a80f-9784767ae3ea-credential-socket" pod "44c38065-cf4a-47fd-a80f-9784767ae3ea" (UID: "44c38065-cf4a-47fd-a80f-9784767ae3ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:31:56.194897 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.194736 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-credential-socket\") pod \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\" (UID: \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\") " Nov 03 14:31:56.194897 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.194800 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44c38065-cf4a-47fd-a80f-9784767ae3ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44c38065-cf4a-47fd-a80f-9784767ae3ea-istio-podinfo\") pod \"44c38065-cf4a-47fd-a80f-9784767ae3ea\" (UID: \"44c38065-cf4a-47fd-a80f-9784767ae3ea\") " Nov 03 14:31:56.195111 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.194916 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-istio-data\") pod \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\" (UID: \"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0\") " Nov 03 14:31:56.195111 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.194921 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-workload-certs" pod "76bc22bf-ed6f-4f6b-a225-5611aa3b73b0" (UID: "76bc22bf-ed6f-4f6b-a225-5611aa3b73b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:31:56.195111 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.194960 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-workload-socket" pod "76bc22bf-ed6f-4f6b-a225-5611aa3b73b0" (UID: "76bc22bf-ed6f-4f6b-a225-5611aa3b73b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:31:56.195111 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.194979 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44c38065-cf4a-47fd-a80f-9784767ae3ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44c38065-cf4a-47fd-a80f-9784767ae3ea-workload-certs\") pod \"44c38065-cf4a-47fd-a80f-9784767ae3ea\" (UID: \"44c38065-cf4a-47fd-a80f-9784767ae3ea\") " Nov 03 14:31:56.195111 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.195006 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44c38065-cf4a-47fd-a80f-9784767ae3ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44c38065-cf4a-47fd-a80f-9784767ae3ea-istiod-ca-cert\") pod \"44c38065-cf4a-47fd-a80f-9784767ae3ea\" (UID: \"44c38065-cf4a-47fd-a80f-9784767ae3ea\") " Nov 03 14:31:56.195111 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.195022 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-istiod-ca-cert" pod "76bc22bf-ed6f-4f6b-a225-5611aa3b73b0" (UID: "76bc22bf-ed6f-4f6b-a225-5611aa3b73b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:31:56.195415 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.195182 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-credential-socket" pod "76bc22bf-ed6f-4f6b-a225-5611aa3b73b0" (UID: "76bc22bf-ed6f-4f6b-a225-5611aa3b73b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:31:56.195415 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.195372 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-istio-data" pod "76bc22bf-ed6f-4f6b-a225-5611aa3b73b0" (UID: "76bc22bf-ed6f-4f6b-a225-5611aa3b73b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:31:56.195532 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.195467 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c38065-cf4a-47fd-a80f-9784767ae3ea-workload-certs" pod "44c38065-cf4a-47fd-a80f-9784767ae3ea" (UID: "44c38065-cf4a-47fd-a80f-9784767ae3ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:31:56.195593 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.195540 5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-istiod-ca-cert\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:31:56.195593 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.195561 5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-workload-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:31:56.195593 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.195575 5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-credential-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:31:56.195593 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.195588 5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-istio-data\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:31:56.195889 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.195603 5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c38065-cf4a-47fd-a80f-9784767ae3ea-workload-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:31:56.195889 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.195619 5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c38065-cf4a-47fd-a80f-9784767ae3ea-credential-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:31:56.195889 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.195633 5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-workload-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:31:56.195889 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.195684 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44c38065-cf4a-47fd-a80f-9784767ae3ea-istiod-ca-cert" pod "44c38065-cf4a-47fd-a80f-9784767ae3ea" (UID: "44c38065-cf4a-47fd-a80f-9784767ae3ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:31:56.197156 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.197127 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44c38065-cf4a-47fd-a80f-9784767ae3ea-istio-podinfo" pod "44c38065-cf4a-47fd-a80f-9784767ae3ea" (UID: "44c38065-cf4a-47fd-a80f-9784767ae3ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:31:56.197457 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.197434 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44c38065-cf4a-47fd-a80f-9784767ae3ea-kube-api-access-lln5r" pod "44c38065-cf4a-47fd-a80f-9784767ae3ea" (UID: "44c38065-cf4a-47fd-a80f-9784767ae3ea"). InnerVolumeSpecName "kube-api-access-lln5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:31:56.197587 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.197567 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c38065-cf4a-47fd-a80f-9784767ae3ea-istio-envoy" pod "44c38065-cf4a-47fd-a80f-9784767ae3ea" (UID: "44c38065-cf4a-47fd-a80f-9784767ae3ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:31:56.197709 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.197687 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-istio-podinfo" pod "76bc22bf-ed6f-4f6b-a225-5611aa3b73b0" (UID: "76bc22bf-ed6f-4f6b-a225-5611aa3b73b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:31:56.197799 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.197772 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-istio-envoy" pod "76bc22bf-ed6f-4f6b-a225-5611aa3b73b0" (UID: "76bc22bf-ed6f-4f6b-a225-5611aa3b73b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:31:56.197936 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.197922 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-kube-api-access-c6fnm" pod "76bc22bf-ed6f-4f6b-a225-5611aa3b73b0" (UID: "76bc22bf-ed6f-4f6b-a225-5611aa3b73b0"). InnerVolumeSpecName "kube-api-access-c6fnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:31:56.198157 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.198129 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-istio-token" pod "76bc22bf-ed6f-4f6b-a225-5611aa3b73b0" (UID: "76bc22bf-ed6f-4f6b-a225-5611aa3b73b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:31:56.296264 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.296221 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44c38065-cf4a-47fd-a80f-9784767ae3ea-istio-token\" (UniqueName: \"kubernetes.io/projected/44c38065-cf4a-47fd-a80f-9784767ae3ea-istio-token\") pod \"44c38065-cf4a-47fd-a80f-9784767ae3ea\" (UID: \"44c38065-cf4a-47fd-a80f-9784767ae3ea\") " Nov 03 14:31:56.296464 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.296323 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44c38065-cf4a-47fd-a80f-9784767ae3ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44c38065-cf4a-47fd-a80f-9784767ae3ea-istio-data\") pod \"44c38065-cf4a-47fd-a80f-9784767ae3ea\" (UID: \"44c38065-cf4a-47fd-a80f-9784767ae3ea\") " Nov 03 14:31:56.296680 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.296656 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c38065-cf4a-47fd-a80f-9784767ae3ea-istio-data" pod "44c38065-cf4a-47fd-a80f-9784767ae3ea" (UID: "44c38065-cf4a-47fd-a80f-9784767ae3ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:31:56.296817 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.296798 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lln5r\" (UniqueName: \"kubernetes.io/projected/44c38065-cf4a-47fd-a80f-9784767ae3ea-kube-api-access-lln5r\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:31:56.297186 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.296823 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6fnm\" (UniqueName: \"kubernetes.io/projected/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-kube-api-access-c6fnm\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:31:56.297186 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.296852 5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-istio-token\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:31:56.297186 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.296865 5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44c38065-cf4a-47fd-a80f-9784767ae3ea-istio-envoy\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:31:56.297186 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.296878 5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-istio-envoy\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:31:56.297186 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.296892 5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44c38065-cf4a-47fd-a80f-9784767ae3ea-istio-data\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:31:56.297186 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.296904 5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0-istio-podinfo\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:31:56.297186 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.296918 5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44c38065-cf4a-47fd-a80f-9784767ae3ea-istio-podinfo\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:31:56.297186 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.296931 5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44c38065-cf4a-47fd-a80f-9784767ae3ea-workload-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:31:56.297186 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.296946 5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44c38065-cf4a-47fd-a80f-9784767ae3ea-istiod-ca-cert\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:31:56.299242 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.299213 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44c38065-cf4a-47fd-a80f-9784767ae3ea-istio-token" pod "44c38065-cf4a-47fd-a80f-9784767ae3ea" (UID: "44c38065-cf4a-47fd-a80f-9784767ae3ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:31:56.398689 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.398604 5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44c38065-cf4a-47fd-a80f-9784767ae3ea-istio-token\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:31:56.492747 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.492710 5060 generic.go:356] "Generic (PLEG): container finished" podID="76bc22bf-ed6f-4f6b-a225-5611aa3b73b0" containerID="84208702fd3a29e1d27ad3e868908ab06f7b863861cf91399568acab9f972c7f" exitCode=0 Nov 03 14:31:56.492967 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.492779 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" Nov 03 14:31:56.492967 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.492793 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" event={"ID":"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0","Type":"ContainerDied","Data":"84208702fd3a29e1d27ad3e868908ab06f7b863861cf91399568acab9f972c7f"} Nov 03 14:31:56.492967 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.492921 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg" event={"ID":"76bc22bf-ed6f-4f6b-a225-5611aa3b73b0","Type":"ContainerDied","Data":"73d985cf26b3be037dc028071e0b31167938d3d999798ef053f8eb28b332d880"} Nov 03 14:31:56.492967 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.492953 5060 scope.go:122] "RemoveContainer" containerID="84208702fd3a29e1d27ad3e868908ab06f7b863861cf91399568acab9f972c7f" Nov 03 14:31:56.494317 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.494296 5060 generic.go:356] "Generic (PLEG): container finished" podID="44c38065-cf4a-47fd-a80f-9784767ae3ea" containerID="3b816520e9f4520b8feb02852c83b1fa3c11327e768a0a39b97bc0b347b3a533" exitCode=0 Nov 03 14:31:56.494416 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.494367 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" event={"ID":"44c38065-cf4a-47fd-a80f-9784767ae3ea","Type":"ContainerDied","Data":"3b816520e9f4520b8feb02852c83b1fa3c11327e768a0a39b97bc0b347b3a533"} Nov 03 14:31:56.494416 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.494373 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" Nov 03 14:31:56.494416 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.494406 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs" event={"ID":"44c38065-cf4a-47fd-a80f-9784767ae3ea","Type":"ContainerDied","Data":"3ac3ca6d84bd068b85541a1da6341ecbc89ec0b2bc65d2e0263732ba35169e7a"} Nov 03 14:31:56.504508 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.504493 5060 scope.go:122] "RemoveContainer" containerID="84208702fd3a29e1d27ad3e868908ab06f7b863861cf91399568acab9f972c7f" Nov 03 14:31:56.504795 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:31:56.504773 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84208702fd3a29e1d27ad3e868908ab06f7b863861cf91399568acab9f972c7f\": container with ID starting with 84208702fd3a29e1d27ad3e868908ab06f7b863861cf91399568acab9f972c7f not found: ID does not exist" containerID="84208702fd3a29e1d27ad3e868908ab06f7b863861cf91399568acab9f972c7f" Nov 03 14:31:56.504922 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.504800 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84208702fd3a29e1d27ad3e868908ab06f7b863861cf91399568acab9f972c7f"} err="failed to get container status \"84208702fd3a29e1d27ad3e868908ab06f7b863861cf91399568acab9f972c7f\": rpc error: code = NotFound desc = could not find container \"84208702fd3a29e1d27ad3e868908ab06f7b863861cf91399568acab9f972c7f\": container with ID starting with 84208702fd3a29e1d27ad3e868908ab06f7b863861cf91399568acab9f972c7f not found: ID does not exist" Nov 03 14:31:56.504922 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.504818 5060 scope.go:122] "RemoveContainer" containerID="3b816520e9f4520b8feb02852c83b1fa3c11327e768a0a39b97bc0b347b3a533" Nov 03 14:31:56.514927 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.514910 5060 scope.go:122] "RemoveContainer" containerID="3b816520e9f4520b8feb02852c83b1fa3c11327e768a0a39b97bc0b347b3a533" Nov 03 14:31:56.515203 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:31:56.515182 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b816520e9f4520b8feb02852c83b1fa3c11327e768a0a39b97bc0b347b3a533\": container with ID starting with 3b816520e9f4520b8feb02852c83b1fa3c11327e768a0a39b97bc0b347b3a533 not found: ID does not exist" containerID="3b816520e9f4520b8feb02852c83b1fa3c11327e768a0a39b97bc0b347b3a533" Nov 03 14:31:56.515255 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.515213 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b816520e9f4520b8feb02852c83b1fa3c11327e768a0a39b97bc0b347b3a533"} err="failed to get container status \"3b816520e9f4520b8feb02852c83b1fa3c11327e768a0a39b97bc0b347b3a533\": rpc error: code = NotFound desc = could not find container \"3b816520e9f4520b8feb02852c83b1fa3c11327e768a0a39b97bc0b347b3a533\": container with ID starting with 3b816520e9f4520b8feb02852c83b1fa3c11327e768a0a39b97bc0b347b3a533 not found: ID does not exist" Nov 03 14:31:56.554782 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.554752 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg"] Nov 03 14:31:56.567341 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.567313 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lpzqg"] Nov 03 14:31:56.605024 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.604990 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs"] Nov 03 14:31:56.612850 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:56.612809 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7lxxs"] Nov 03 14:31:57.044152 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:57.044118 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf1bc777-cc6qj"] Nov 03 14:31:57.052585 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:57.052559 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf1bc777-cc6qj"] Nov 03 14:31:58.251629 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:58.251597 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0229f45e-0f1c-42dd-b0bc-27b15a88392b" path="/var/lib/kubelet/pods/0229f45e-0f1c-42dd-b0bc-27b15a88392b/volumes" Nov 03 14:31:58.252340 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:58.252316 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44c38065-cf4a-47fd-a80f-9784767ae3ea" path="/var/lib/kubelet/pods/44c38065-cf4a-47fd-a80f-9784767ae3ea/volumes" Nov 03 14:31:58.252937 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:31:58.252910 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76bc22bf-ed6f-4f6b-a225-5611aa3b73b0" path="/var/lib/kubelet/pods/76bc22bf-ed6f-4f6b-a225-5611aa3b73b0/volumes" Nov 03 14:32:00.465268 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:00.465225 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p"] Nov 03 14:32:00.465795 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:00.465780 5060 cpu_manager.go:414] "containerMap: removing container" podUID="76bc22bf-ed6f-4f6b-a225-5611aa3b73b0" containerName="istio-proxy" Nov 03 14:32:00.465795 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:00.465795 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76bc22bf-ed6f-4f6b-a225-5611aa3b73b0" containerName="istio-proxy" Nov 03 14:32:00.465903 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:00.465820 5060 cpu_manager.go:414] "containerMap: removing container" podUID="881ff6f4-04c1-471e-8ade-1c61fe6486a4" containerName="revoke-keys" Nov 03 14:32:00.465903 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:00.465826 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="881ff6f4-04c1-471e-8ade-1c61fe6486a4" containerName="revoke-keys" Nov 03 14:32:00.465903 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:00.465850 5060 cpu_manager.go:414] "containerMap: removing container" podUID="44c38065-cf4a-47fd-a80f-9784767ae3ea" containerName="istio-proxy" Nov 03 14:32:00.465903 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:00.465856 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44c38065-cf4a-47fd-a80f-9784767ae3ea" containerName="istio-proxy" Nov 03 14:32:00.466025 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:00.465932 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="881ff6f4-04c1-471e-8ade-1c61fe6486a4" containerName="revoke-keys" Nov 03 14:32:00.466025 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:00.465942 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="44c38065-cf4a-47fd-a80f-9784767ae3ea" containerName="istio-proxy" Nov 03 14:32:00.466025 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:00.465950 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="76bc22bf-ed6f-4f6b-a225-5611aa3b73b0" containerName="istio-proxy" Nov 03 14:32:00.470884 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:00.470865 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p" Nov 03 14:32:00.493741 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:00.493713 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p"] Nov 03 14:32:00.502883 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:00.502860 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:32:00.503262 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:00.503242 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-4a67e25f-serving-cert\"" Nov 03 14:32:00.503598 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:00.503574 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7cpjn\"" Nov 03 14:32:00.503598 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:00.503428 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:32:00.503808 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:00.503527 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 14:32:00.552996 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:00.552962 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9k8p\" (UniqueName: \"kubernetes.io/projected/393199fc-cbb3-4681-a6ee-cce53def4689-kube-api-access-d9k8p\") pod \"maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p\" (UID: \"393199fc-cbb3-4681-a6ee-cce53def4689\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p" Nov 03 14:32:00.553193 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:00.553016 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/393199fc-cbb3-4681-a6ee-cce53def4689-openshift-service-ca\") pod \"maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p\" (UID: \"393199fc-cbb3-4681-a6ee-cce53def4689\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p" Nov 03 14:32:00.553193 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:00.553132 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/393199fc-cbb3-4681-a6ee-cce53def4689-metrics-certs\") pod \"maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p\" (UID: \"393199fc-cbb3-4681-a6ee-cce53def4689\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p" Nov 03 14:32:00.553345 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:00.553322 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/393199fc-cbb3-4681-a6ee-cce53def4689-maas-api-tls\") pod \"maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p\" (UID: \"393199fc-cbb3-4681-a6ee-cce53def4689\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p" Nov 03 14:32:00.654807 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:00.654772 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9k8p\" (UniqueName: \"kubernetes.io/projected/393199fc-cbb3-4681-a6ee-cce53def4689-kube-api-access-d9k8p\") pod \"maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p\" (UID: \"393199fc-cbb3-4681-a6ee-cce53def4689\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p" Nov 03 14:32:00.654807 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:00.654814 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/393199fc-cbb3-4681-a6ee-cce53def4689-openshift-service-ca\") pod \"maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p\" (UID: \"393199fc-cbb3-4681-a6ee-cce53def4689\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p" Nov 03 14:32:00.655096 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:00.654868 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/393199fc-cbb3-4681-a6ee-cce53def4689-metrics-certs\") pod \"maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p\" (UID: \"393199fc-cbb3-4681-a6ee-cce53def4689\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p" Nov 03 14:32:00.655096 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:00.654945 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/393199fc-cbb3-4681-a6ee-cce53def4689-maas-api-tls\") pod \"maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p\" (UID: \"393199fc-cbb3-4681-a6ee-cce53def4689\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p" Nov 03 14:32:00.655223 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:32:00.655098 5060 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-serving-cert: secret "maas-api-e2e-mt-4a67e25f-serving-cert" not found Nov 03 14:32:00.655223 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:32:00.655162 5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/393199fc-cbb3-4681-a6ee-cce53def4689-maas-api-tls podName:393199fc-cbb3-4681-a6ee-cce53def4689 nodeName:}" failed. No retries permitted until 2026-11-03 14:32:01.155147527 +0000 UTC m=+1255.043937087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/393199fc-cbb3-4681-a6ee-cce53def4689-maas-api-tls") pod "maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p" (UID: "393199fc-cbb3-4681-a6ee-cce53def4689") : secret "maas-api-e2e-mt-4a67e25f-serving-cert" not found Nov 03 14:32:00.655599 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:00.655571 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/393199fc-cbb3-4681-a6ee-cce53def4689-openshift-service-ca\") pod \"maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p\" (UID: \"393199fc-cbb3-4681-a6ee-cce53def4689\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p" Nov 03 14:32:00.657325 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:00.657305 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/393199fc-cbb3-4681-a6ee-cce53def4689-metrics-certs\") pod \"maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p\" (UID: \"393199fc-cbb3-4681-a6ee-cce53def4689\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p" Nov 03 14:32:00.679626 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:00.679579 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9k8p\" (UniqueName: \"kubernetes.io/projected/393199fc-cbb3-4681-a6ee-cce53def4689-kube-api-access-d9k8p\") pod \"maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p\" (UID: \"393199fc-cbb3-4681-a6ee-cce53def4689\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p" Nov 03 14:32:01.162858 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:01.162769 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/393199fc-cbb3-4681-a6ee-cce53def4689-maas-api-tls\") pod \"maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p\" (UID: \"393199fc-cbb3-4681-a6ee-cce53def4689\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p" Nov 03 14:32:01.233480 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:01.233446 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/393199fc-cbb3-4681-a6ee-cce53def4689-maas-api-tls\") pod \"maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p\" (UID: \"393199fc-cbb3-4681-a6ee-cce53def4689\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p" Nov 03 14:32:01.384549 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:01.384513 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p" Nov 03 14:32:01.710536 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:01.710501 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p"] Nov 03 14:32:01.727928 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:01.727901 5060 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 14:32:02.560683 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:02.560576 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p" event={"ID":"393199fc-cbb3-4681-a6ee-cce53def4689","Type":"ContainerStarted","Data":"1da03098efefbe5bc81458ba4d9cffdaea71f210cf841d738a7edafa784e68c9"} Nov 03 14:32:04.337858 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:04.333150 5060 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:32:04.337858 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:04.333234 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" podUID="b671512d-59b5-4ddd-959d-1d9150d41268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:32:05.977933 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:05.977818 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p" event={"ID":"393199fc-cbb3-4681-a6ee-cce53def4689","Type":"ContainerStarted","Data":"ca828d6ecf85ab2bfd1ea064cfafe4bd2ba5909fa0de60b6432c2a4af19a5faf"} Nov 03 14:32:05.977933 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:05.977914 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p" Nov 03 14:32:06.046496 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:06.046440 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p" podStartSLOduration=2.070386235 podStartE2EDuration="6.046426714s" totalImagesPullingTime="3.976040479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:32:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:32:06.04586565 +0000 UTC m=+1259.934655231" watchObservedRunningTime="2026-11-03 14:32:06.046426714 +0000 UTC m=+1259.935216295" Nov 03 14:32:07.041867 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:07.041800 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9759f85a-lghlg"] Nov 03 14:32:07.053024 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:07.052997 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9759f85a-lghlg"] Nov 03 14:32:07.203342 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:07.203306 5060 scope.go:122] "RemoveContainer" containerID="d228f7837b3cbe360b8d4af736e9ef3b60f7ded190cd0fd547f4c7d6dea560a0" Nov 03 14:32:07.214080 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:07.214056 5060 scope.go:122] "RemoveContainer" containerID="e3e8ed9dc643fac1c288c6330e254fb3152cda3b4dcb8df68efa6cb5ab0c4aca" Nov 03 14:32:07.449490 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:07.449396 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-09d577dc-6c6fc7b44b-59zxt"] Nov 03 14:32:07.455019 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:07.454997 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d577dc-6c6fc7b44b-59zxt" Nov 03 14:32:07.476658 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:07.476634 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-09d577dc-6c6fc7b44b-59zxt"] Nov 03 14:32:07.480914 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:07.480885 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34e15205-cba4-409a-abb1-1712d2292684-openshift-service-ca\") pod \"maas-api-e2e-ait-09d577dc-6c6fc7b44b-59zxt\" (UID: \"34e15205-cba4-409a-abb1-1712d2292684\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d577dc-6c6fc7b44b-59zxt" Nov 03 14:32:07.481052 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:07.480937 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2hn7\" (UniqueName: \"kubernetes.io/projected/34e15205-cba4-409a-abb1-1712d2292684-kube-api-access-r2hn7\") pod \"maas-api-e2e-ait-09d577dc-6c6fc7b44b-59zxt\" (UID: \"34e15205-cba4-409a-abb1-1712d2292684\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d577dc-6c6fc7b44b-59zxt" Nov 03 14:32:07.481117 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:07.481057 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34e15205-cba4-409a-abb1-1712d2292684-maas-api-tls\") pod \"maas-api-e2e-ait-09d577dc-6c6fc7b44b-59zxt\" (UID: \"34e15205-cba4-409a-abb1-1712d2292684\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d577dc-6c6fc7b44b-59zxt" Nov 03 14:32:07.481273 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:07.481256 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34e15205-cba4-409a-abb1-1712d2292684-metrics-certs\") pod \"maas-api-e2e-ait-09d577dc-6c6fc7b44b-59zxt\" (UID: \"34e15205-cba4-409a-abb1-1712d2292684\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d577dc-6c6fc7b44b-59zxt" Nov 03 14:32:07.487498 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:07.487466 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-09d577dc-serving-cert\"" Nov 03 14:32:07.582589 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:07.582551 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34e15205-cba4-409a-abb1-1712d2292684-metrics-certs\") pod \"maas-api-e2e-ait-09d577dc-6c6fc7b44b-59zxt\" (UID: \"34e15205-cba4-409a-abb1-1712d2292684\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d577dc-6c6fc7b44b-59zxt" Nov 03 14:32:07.582770 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:07.582618 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34e15205-cba4-409a-abb1-1712d2292684-openshift-service-ca\") pod \"maas-api-e2e-ait-09d577dc-6c6fc7b44b-59zxt\" (UID: \"34e15205-cba4-409a-abb1-1712d2292684\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d577dc-6c6fc7b44b-59zxt" Nov 03 14:32:07.582770 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:07.582640 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2hn7\" (UniqueName: \"kubernetes.io/projected/34e15205-cba4-409a-abb1-1712d2292684-kube-api-access-r2hn7\") pod \"maas-api-e2e-ait-09d577dc-6c6fc7b44b-59zxt\" (UID: \"34e15205-cba4-409a-abb1-1712d2292684\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d577dc-6c6fc7b44b-59zxt" Nov 03 14:32:07.582770 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:07.582675 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34e15205-cba4-409a-abb1-1712d2292684-maas-api-tls\") pod \"maas-api-e2e-ait-09d577dc-6c6fc7b44b-59zxt\" (UID: \"34e15205-cba4-409a-abb1-1712d2292684\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d577dc-6c6fc7b44b-59zxt" Nov 03 14:32:07.583447 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:07.583421 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34e15205-cba4-409a-abb1-1712d2292684-openshift-service-ca\") pod \"maas-api-e2e-ait-09d577dc-6c6fc7b44b-59zxt\" (UID: \"34e15205-cba4-409a-abb1-1712d2292684\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d577dc-6c6fc7b44b-59zxt" Nov 03 14:32:07.585241 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:07.585211 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34e15205-cba4-409a-abb1-1712d2292684-metrics-certs\") pod \"maas-api-e2e-ait-09d577dc-6c6fc7b44b-59zxt\" (UID: \"34e15205-cba4-409a-abb1-1712d2292684\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d577dc-6c6fc7b44b-59zxt" Nov 03 14:32:07.585241 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:07.585231 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34e15205-cba4-409a-abb1-1712d2292684-maas-api-tls\") pod \"maas-api-e2e-ait-09d577dc-6c6fc7b44b-59zxt\" (UID: \"34e15205-cba4-409a-abb1-1712d2292684\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d577dc-6c6fc7b44b-59zxt" Nov 03 14:32:07.605380 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:07.605352 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2hn7\" (UniqueName: \"kubernetes.io/projected/34e15205-cba4-409a-abb1-1712d2292684-kube-api-access-r2hn7\") pod \"maas-api-e2e-ait-09d577dc-6c6fc7b44b-59zxt\" (UID: \"34e15205-cba4-409a-abb1-1712d2292684\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d577dc-6c6fc7b44b-59zxt" Nov 03 14:32:07.768473 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:07.768439 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d577dc-6c6fc7b44b-59zxt" Nov 03 14:32:08.075508 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:08.075476 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-09d577dc-6c6fc7b44b-59zxt"] Nov 03 14:32:08.077008 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:32:08.076985 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34e15205_cba4_409a_abb1_1712d2292684.slice/crio-69cf8a65820affcd09191b3317aacb317017956dcf88ea6908d71b1ef9720a92 WatchSource:0}: Error finding container 69cf8a65820affcd09191b3317aacb317017956dcf88ea6908d71b1ef9720a92: Status 404 returned error can't find the container with id 69cf8a65820affcd09191b3317aacb317017956dcf88ea6908d71b1ef9720a92 Nov 03 14:32:08.249017 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:08.248977 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="054d8392-289a-4eac-8926-ee85aaf15395" path="/var/lib/kubelet/pods/054d8392-289a-4eac-8926-ee85aaf15395/volumes" Nov 03 14:32:09.001366 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:09.001333 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d577dc-6c6fc7b44b-59zxt" event={"ID":"34e15205-cba4-409a-abb1-1712d2292684","Type":"ContainerStarted","Data":"b5e4e668ece8b382954b91b8811cf1b0477d8eab7d0fedcf4404094365c29c35"} Nov 03 14:32:09.001519 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:09.001383 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d577dc-6c6fc7b44b-59zxt" event={"ID":"34e15205-cba4-409a-abb1-1712d2292684","Type":"ContainerStarted","Data":"69cf8a65820affcd09191b3317aacb317017956dcf88ea6908d71b1ef9720a92"} Nov 03 14:32:09.001579 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:09.001515 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d577dc-6c6fc7b44b-59zxt" Nov 03 14:32:09.049456 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:09.049402 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d577dc-6c6fc7b44b-59zxt" podStartSLOduration=1.666241071 podStartE2EDuration="2.049389116s" totalImagesPullingTime="383.148045ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:32:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:32:09.045890704 +0000 UTC m=+1262.934680285" watchObservedRunningTime="2026-11-03 14:32:09.049389116 +0000 UTC m=+1262.938178705" Nov 03 14:32:09.441433 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:32:09.441352 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:32:11.989600 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:11.989572 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p" Nov 03 14:32:15.013950 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:15.013919 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d577dc-6c6fc7b44b-59zxt" Nov 03 14:32:17.896858 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:17.887350 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-6653eb-59f9f85797-68tkt"] Nov 03 14:32:17.897358 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:17.897078 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6653eb-59f9f85797-68tkt" Nov 03 14:32:17.930859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:17.917034 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-6653eb-59f9f85797-68tkt"] Nov 03 14:32:17.930859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:17.917066 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-6653eb-serving-cert\"" Nov 03 14:32:18.091336 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:18.091292 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74a86163-4d09-49e1-945d-521f611bda7e-maas-api-tls\") pod \"maas-api-e2e-autores-6653eb-59f9f85797-68tkt\" (UID: \"74a86163-4d09-49e1-945d-521f611bda7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6653eb-59f9f85797-68tkt" Nov 03 14:32:18.123151 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:18.123096 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74a86163-4d09-49e1-945d-521f611bda7e-openshift-service-ca\") pod \"maas-api-e2e-autores-6653eb-59f9f85797-68tkt\" (UID: \"74a86163-4d09-49e1-945d-521f611bda7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6653eb-59f9f85797-68tkt" Nov 03 14:32:18.123508 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:18.123481 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74a86163-4d09-49e1-945d-521f611bda7e-metrics-certs\") pod \"maas-api-e2e-autores-6653eb-59f9f85797-68tkt\" (UID: \"74a86163-4d09-49e1-945d-521f611bda7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6653eb-59f9f85797-68tkt" Nov 03 14:32:18.123759 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:18.123738 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcwmc\" (UniqueName: \"kubernetes.io/projected/74a86163-4d09-49e1-945d-521f611bda7e-kube-api-access-pcwmc\") pod \"maas-api-e2e-autores-6653eb-59f9f85797-68tkt\" (UID: \"74a86163-4d09-49e1-945d-521f611bda7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6653eb-59f9f85797-68tkt" Nov 03 14:32:18.276859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:18.269941 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74a86163-4d09-49e1-945d-521f611bda7e-maas-api-tls\") pod \"maas-api-e2e-autores-6653eb-59f9f85797-68tkt\" (UID: \"74a86163-4d09-49e1-945d-521f611bda7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6653eb-59f9f85797-68tkt" Nov 03 14:32:18.276859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:18.270128 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74a86163-4d09-49e1-945d-521f611bda7e-openshift-service-ca\") pod \"maas-api-e2e-autores-6653eb-59f9f85797-68tkt\" (UID: \"74a86163-4d09-49e1-945d-521f611bda7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6653eb-59f9f85797-68tkt" Nov 03 14:32:18.276859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:18.270163 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74a86163-4d09-49e1-945d-521f611bda7e-metrics-certs\") pod \"maas-api-e2e-autores-6653eb-59f9f85797-68tkt\" (UID: \"74a86163-4d09-49e1-945d-521f611bda7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6653eb-59f9f85797-68tkt" Nov 03 14:32:18.276859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:18.270243 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcwmc\" (UniqueName: \"kubernetes.io/projected/74a86163-4d09-49e1-945d-521f611bda7e-kube-api-access-pcwmc\") pod \"maas-api-e2e-autores-6653eb-59f9f85797-68tkt\" (UID: \"74a86163-4d09-49e1-945d-521f611bda7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6653eb-59f9f85797-68tkt" Nov 03 14:32:18.276859 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:32:18.270775 5060 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-6653eb-serving-cert: secret "maas-api-e2e-autores-6653eb-serving-cert" not found Nov 03 14:32:18.276859 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:32:18.270885 5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74a86163-4d09-49e1-945d-521f611bda7e-maas-api-tls podName:74a86163-4d09-49e1-945d-521f611bda7e nodeName:}" failed. No retries permitted until 2026-11-03 14:32:18.770861048 +0000 UTC m=+1272.659650641 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/74a86163-4d09-49e1-945d-521f611bda7e-maas-api-tls") pod "maas-api-e2e-autores-6653eb-59f9f85797-68tkt" (UID: "74a86163-4d09-49e1-945d-521f611bda7e") : secret "maas-api-e2e-autores-6653eb-serving-cert" not found Nov 03 14:32:18.276859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:18.271825 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74a86163-4d09-49e1-945d-521f611bda7e-openshift-service-ca\") pod \"maas-api-e2e-autores-6653eb-59f9f85797-68tkt\" (UID: \"74a86163-4d09-49e1-945d-521f611bda7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6653eb-59f9f85797-68tkt" Nov 03 14:32:18.529913 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:18.515319 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74a86163-4d09-49e1-945d-521f611bda7e-metrics-certs\") pod \"maas-api-e2e-autores-6653eb-59f9f85797-68tkt\" (UID: \"74a86163-4d09-49e1-945d-521f611bda7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6653eb-59f9f85797-68tkt" Nov 03 14:32:18.598870 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:18.598510 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcwmc\" (UniqueName: \"kubernetes.io/projected/74a86163-4d09-49e1-945d-521f611bda7e-kube-api-access-pcwmc\") pod \"maas-api-e2e-autores-6653eb-59f9f85797-68tkt\" (UID: \"74a86163-4d09-49e1-945d-521f611bda7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6653eb-59f9f85797-68tkt" Nov 03 14:32:18.802925 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:18.801162 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74a86163-4d09-49e1-945d-521f611bda7e-maas-api-tls\") pod \"maas-api-e2e-autores-6653eb-59f9f85797-68tkt\" (UID: \"74a86163-4d09-49e1-945d-521f611bda7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6653eb-59f9f85797-68tkt" Nov 03 14:32:18.802925 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:32:18.801460 5060 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-6653eb-serving-cert: secret "maas-api-e2e-autores-6653eb-serving-cert" not found Nov 03 14:32:18.802925 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:32:18.801527 5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74a86163-4d09-49e1-945d-521f611bda7e-maas-api-tls podName:74a86163-4d09-49e1-945d-521f611bda7e nodeName:}" failed. No retries permitted until 2026-11-03 14:32:19.801508776 +0000 UTC m=+1273.690298355 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/74a86163-4d09-49e1-945d-521f611bda7e-maas-api-tls") pod "maas-api-e2e-autores-6653eb-59f9f85797-68tkt" (UID: "74a86163-4d09-49e1-945d-521f611bda7e") : secret "maas-api-e2e-autores-6653eb-serving-cert" not found Nov 03 14:32:19.835081 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:19.835039 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74a86163-4d09-49e1-945d-521f611bda7e-maas-api-tls\") pod \"maas-api-e2e-autores-6653eb-59f9f85797-68tkt\" (UID: \"74a86163-4d09-49e1-945d-521f611bda7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6653eb-59f9f85797-68tkt" Nov 03 14:32:19.837880 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:19.837819 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74a86163-4d09-49e1-945d-521f611bda7e-maas-api-tls\") pod \"maas-api-e2e-autores-6653eb-59f9f85797-68tkt\" (UID: \"74a86163-4d09-49e1-945d-521f611bda7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6653eb-59f9f85797-68tkt" Nov 03 14:32:20.129363 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:20.129266 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6653eb-59f9f85797-68tkt" Nov 03 14:32:20.442333 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:20.442184 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-6653eb-59f9f85797-68tkt"] Nov 03 14:32:20.446073 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:32:20.446045 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74a86163_4d09_49e1_945d_521f611bda7e.slice/crio-b573f04cd163cf8a3b9699024391a05e55168135b5e1cbda597eba1798baaa47 WatchSource:0}: Error finding container b573f04cd163cf8a3b9699024391a05e55168135b5e1cbda597eba1798baaa47: Status 404 returned error can't find the container with id b573f04cd163cf8a3b9699024391a05e55168135b5e1cbda597eba1798baaa47 Nov 03 14:32:20.576060 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:20.576005 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6653eb-59f9f85797-68tkt" event={"ID":"74a86163-4d09-49e1-945d-521f611bda7e","Type":"ContainerStarted","Data":"b573f04cd163cf8a3b9699024391a05e55168135b5e1cbda597eba1798baaa47"} Nov 03 14:32:21.584635 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:21.584600 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6653eb-59f9f85797-68tkt" event={"ID":"74a86163-4d09-49e1-945d-521f611bda7e","Type":"ContainerStarted","Data":"e8f36f62e2407ecb1636fc01fe6cba2f786f75cd70dc3c225a7b923fc4199eb3"} Nov 03 14:32:21.585045 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:21.584735 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6653eb-59f9f85797-68tkt" Nov 03 14:32:21.635878 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:21.627808 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6653eb-59f9f85797-68tkt" podStartSLOduration=4.310074153 podStartE2EDuration="4.6277887s" totalImagesPullingTime="317.714547ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:32:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:32:21.627229216 +0000 UTC m=+1275.516018800" watchObservedRunningTime="2026-11-03 14:32:21.6277887 +0000 UTC m=+1275.516578281" Nov 03 14:32:25.055697 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:25.055656 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc0cd4ad-vc5q8"] Nov 03 14:32:25.068384 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:25.068350 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc0cd4ad-vc5q8"] Nov 03 14:32:26.249714 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:26.249683 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b011271-14e8-41f9-b8e4-e978dbdacc27" path="/var/lib/kubelet/pods/7b011271-14e8-41f9-b8e4-e978dbdacc27/volumes" Nov 03 14:32:27.597563 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:27.597528 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6653eb-59f9f85797-68tkt" Nov 03 14:32:29.644873 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:29.644826 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7319-2f122b"] Nov 03 14:32:29.652135 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:29.652112 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7319-2f122b" Nov 03 14:32:29.668927 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:29.668901 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7319-2f122b"] Nov 03 14:32:29.677359 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:29.677333 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-9sm86\"" Nov 03 14:32:29.763929 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:29.763883 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7pz5\" (UniqueName: \"kubernetes.io/projected/d35f56d4-7270-41dc-aff5-1acb41e019e0-kube-api-access-z7pz5\") pod \"test-curl-7319-2f122b\" (UID: \"d35f56d4-7270-41dc-aff5-1acb41e019e0\") " pod="opendatahub/test-curl-7319-2f122b" Nov 03 14:32:29.865347 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:29.865315 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7pz5\" (UniqueName: \"kubernetes.io/projected/d35f56d4-7270-41dc-aff5-1acb41e019e0-kube-api-access-z7pz5\") pod \"test-curl-7319-2f122b\" (UID: \"d35f56d4-7270-41dc-aff5-1acb41e019e0\") " pod="opendatahub/test-curl-7319-2f122b" Nov 03 14:32:29.887762 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:29.887730 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7pz5\" (UniqueName: \"kubernetes.io/projected/d35f56d4-7270-41dc-aff5-1acb41e019e0-kube-api-access-z7pz5\") pod \"test-curl-7319-2f122b\" (UID: \"d35f56d4-7270-41dc-aff5-1acb41e019e0\") " pod="opendatahub/test-curl-7319-2f122b" Nov 03 14:32:29.965331 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:29.965254 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7319-2f122b" Nov 03 14:32:30.275446 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:30.275419 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7319-2f122b"] Nov 03 14:32:30.276860 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:32:30.276823 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd35f56d4_7270_41dc_aff5_1acb41e019e0.slice/crio-dcc0c74221779a822e31bb080322bc166538204f7f7f56873f609470d1695ea9 WatchSource:0}: Error finding container dcc0c74221779a822e31bb080322bc166538204f7f7f56873f609470d1695ea9: Status 404 returned error can't find the container with id dcc0c74221779a822e31bb080322bc166538204f7f7f56873f609470d1695ea9 Nov 03 14:32:30.658406 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:30.658323 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7319-2f122b" event={"ID":"d35f56d4-7270-41dc-aff5-1acb41e019e0","Type":"ContainerStarted","Data":"18d2f3d95ab787844969d29d7c66aa53f85512b01fc839b3eacc32ac8db62689"} Nov 03 14:32:30.658406 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:30.658368 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7319-2f122b" event={"ID":"d35f56d4-7270-41dc-aff5-1acb41e019e0","Type":"ContainerStarted","Data":"dcc0c74221779a822e31bb080322bc166538204f7f7f56873f609470d1695ea9"} Nov 03 14:32:30.703191 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:30.703135 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7319-2f122b" podStartSLOduration=1.7031208329999998 podStartE2EDuration="1.703120833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:32:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:32:30.699349361 +0000 UTC m=+1284.588138964" watchObservedRunningTime="2026-11-03 14:32:30.703120833 +0000 UTC m=+1284.591910414" Nov 03 14:32:31.534969 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:31.534935 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7319-2f122b"] Nov 03 14:32:31.539344 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:31.539308 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7319-2f122b"] Nov 03 14:32:32.676526 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:32.676458 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7319-2f122b" podUID="d35f56d4-7270-41dc-aff5-1acb41e019e0" containerName="test-curl-7319-2f122b" containerID="cri-o://18d2f3d95ab787844969d29d7c66aa53f85512b01fc839b3eacc32ac8db62689" gracePeriod=2 Nov 03 14:32:34.911496 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:34.911466 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7319-2f122b" Nov 03 14:32:35.040900 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:35.040867 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d35f56d4-7270-41dc-aff5-1acb41e019e0-kube-api-access-z7pz5\" (UniqueName: \"kubernetes.io/projected/d35f56d4-7270-41dc-aff5-1acb41e019e0-kube-api-access-z7pz5\") pod \"d35f56d4-7270-41dc-aff5-1acb41e019e0\" (UID: \"d35f56d4-7270-41dc-aff5-1acb41e019e0\") " Nov 03 14:32:35.042938 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:35.042905 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d35f56d4-7270-41dc-aff5-1acb41e019e0-kube-api-access-z7pz5" pod "d35f56d4-7270-41dc-aff5-1acb41e019e0" (UID: "d35f56d4-7270-41dc-aff5-1acb41e019e0"). InnerVolumeSpecName "kube-api-access-z7pz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:32:35.142951 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:35.142919 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7pz5\" (UniqueName: \"kubernetes.io/projected/d35f56d4-7270-41dc-aff5-1acb41e019e0-kube-api-access-z7pz5\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:32:35.697147 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:35.697110 5060 generic.go:356] "Generic (PLEG): container finished" podID="d35f56d4-7270-41dc-aff5-1acb41e019e0" containerID="18d2f3d95ab787844969d29d7c66aa53f85512b01fc839b3eacc32ac8db62689" exitCode=137 Nov 03 14:32:35.697458 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:35.697168 5060 scope.go:122] "RemoveContainer" containerID="18d2f3d95ab787844969d29d7c66aa53f85512b01fc839b3eacc32ac8db62689" Nov 03 14:32:35.697458 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:35.697220 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7319-2f122b" Nov 03 14:32:35.707175 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:35.707151 5060 scope.go:122] "RemoveContainer" containerID="18d2f3d95ab787844969d29d7c66aa53f85512b01fc839b3eacc32ac8db62689" Nov 03 14:32:35.707429 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:32:35.707409 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18d2f3d95ab787844969d29d7c66aa53f85512b01fc839b3eacc32ac8db62689\": container with ID starting with 18d2f3d95ab787844969d29d7c66aa53f85512b01fc839b3eacc32ac8db62689 not found: ID does not exist" containerID="18d2f3d95ab787844969d29d7c66aa53f85512b01fc839b3eacc32ac8db62689" Nov 03 14:32:35.707484 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:35.707438 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18d2f3d95ab787844969d29d7c66aa53f85512b01fc839b3eacc32ac8db62689"} err="failed to get container status \"18d2f3d95ab787844969d29d7c66aa53f85512b01fc839b3eacc32ac8db62689\": rpc error: code = NotFound desc = could not find container \"18d2f3d95ab787844969d29d7c66aa53f85512b01fc839b3eacc32ac8db62689\": container with ID starting with 18d2f3d95ab787844969d29d7c66aa53f85512b01fc839b3eacc32ac8db62689 not found: ID does not exist" Nov 03 14:32:35.720175 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:32:35.720148 5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7319-2f122b\" is forbidden: User \"system:node:ip-10-0-143-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-171.ec2.internal' and this object" podUID="d35f56d4-7270-41dc-aff5-1acb41e019e0" pod="opendatahub/test-curl-7319-2f122b" Nov 03 14:32:36.252058 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:36.252026 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d35f56d4-7270-41dc-aff5-1acb41e019e0" path="/var/lib/kubelet/pods/d35f56d4-7270-41dc-aff5-1acb41e019e0/volumes" Nov 03 14:32:36.262183 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:32:36.262149 5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7319-2f122b\" is forbidden: User \"system:node:ip-10-0-143-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-171.ec2.internal' and this object" podUID="d35f56d4-7270-41dc-aff5-1acb41e019e0" pod="opendatahub/test-curl-7319-2f122b" Nov 03 14:32:37.037028 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:37.036995 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-4a67e25f/e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p"] Nov 03 14:32:39.442183 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:32:39.442138 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:32:46.161536 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:46.161497 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9"] Nov 03 14:32:46.162113 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:46.162097 5060 cpu_manager.go:414] "containerMap: removing container" podUID="d35f56d4-7270-41dc-aff5-1acb41e019e0" containerName="test-curl-7319-2f122b" Nov 03 14:32:46.162164 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:46.162114 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d35f56d4-7270-41dc-aff5-1acb41e019e0" containerName="test-curl-7319-2f122b" Nov 03 14:32:46.162210 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:46.162201 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d35f56d4-7270-41dc-aff5-1acb41e019e0" containerName="test-curl-7319-2f122b" Nov 03 14:32:46.166983 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:46.166956 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" Nov 03 14:32:46.193584 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:46.193555 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9"] Nov 03 14:32:46.271176 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:46.271140 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a48c702-25f2-4601-9257-8281d6735a35-istio-token\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9\" (UID: \"9a48c702-25f2-4601-9257-8281d6735a35\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" Nov 03 14:32:46.271176 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:46.271181 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a48c702-25f2-4601-9257-8281d6735a35-istio-podinfo\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9\" (UID: \"9a48c702-25f2-4601-9257-8281d6735a35\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" Nov 03 14:32:46.271390 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:46.271318 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a48c702-25f2-4601-9257-8281d6735a35-istio-data\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9\" (UID: \"9a48c702-25f2-4601-9257-8281d6735a35\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" Nov 03 14:32:46.271429 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:46.271391 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a48c702-25f2-4601-9257-8281d6735a35-workload-certs\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9\" (UID: \"9a48c702-25f2-4601-9257-8281d6735a35\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" Nov 03 14:32:46.271429 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:46.271417 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a48c702-25f2-4601-9257-8281d6735a35-credential-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9\" (UID: \"9a48c702-25f2-4601-9257-8281d6735a35\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" Nov 03 14:32:46.271506 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:46.271473 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a48c702-25f2-4601-9257-8281d6735a35-istio-envoy\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9\" (UID: \"9a48c702-25f2-4601-9257-8281d6735a35\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" Nov 03 14:32:46.271629 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:46.271606 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a48c702-25f2-4601-9257-8281d6735a35-workload-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9\" (UID: \"9a48c702-25f2-4601-9257-8281d6735a35\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" Nov 03 14:32:46.271752 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:46.271730 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a48c702-25f2-4601-9257-8281d6735a35-istiod-ca-cert\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9\" (UID: \"9a48c702-25f2-4601-9257-8281d6735a35\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" Nov 03 14:32:46.271823 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:46.271807 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdg6g\" (UniqueName: \"kubernetes.io/projected/9a48c702-25f2-4601-9257-8281d6735a35-kube-api-access-vdg6g\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9\" (UID: \"9a48c702-25f2-4601-9257-8281d6735a35\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" Nov 03 14:32:46.373487 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:46.373456 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a48c702-25f2-4601-9257-8281d6735a35-istiod-ca-cert\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9\" (UID: \"9a48c702-25f2-4601-9257-8281d6735a35\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" Nov 03 14:32:46.373689 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:46.373498 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdg6g\" (UniqueName: \"kubernetes.io/projected/9a48c702-25f2-4601-9257-8281d6735a35-kube-api-access-vdg6g\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9\" (UID: \"9a48c702-25f2-4601-9257-8281d6735a35\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" Nov 03 14:32:46.373689 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:46.373538 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a48c702-25f2-4601-9257-8281d6735a35-istio-token\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9\" (UID: \"9a48c702-25f2-4601-9257-8281d6735a35\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" Nov 03 14:32:46.373689 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:46.373566 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a48c702-25f2-4601-9257-8281d6735a35-istio-podinfo\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9\" (UID: \"9a48c702-25f2-4601-9257-8281d6735a35\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" Nov 03 14:32:46.373689 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:46.373609 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a48c702-25f2-4601-9257-8281d6735a35-istio-data\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9\" (UID: \"9a48c702-25f2-4601-9257-8281d6735a35\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" Nov 03 14:32:46.373689 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:46.373684 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a48c702-25f2-4601-9257-8281d6735a35-workload-certs\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9\" (UID: \"9a48c702-25f2-4601-9257-8281d6735a35\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" Nov 03 14:32:46.373990 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:46.373714 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a48c702-25f2-4601-9257-8281d6735a35-credential-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9\" (UID: \"9a48c702-25f2-4601-9257-8281d6735a35\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" Nov 03 14:32:46.373990 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:46.373756 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a48c702-25f2-4601-9257-8281d6735a35-istio-envoy\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9\" (UID: \"9a48c702-25f2-4601-9257-8281d6735a35\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" Nov 03 14:32:46.373990 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:46.373818 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a48c702-25f2-4601-9257-8281d6735a35-workload-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9\" (UID: \"9a48c702-25f2-4601-9257-8281d6735a35\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" Nov 03 14:32:46.374154 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:46.374123 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a48c702-25f2-4601-9257-8281d6735a35-istio-data\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9\" (UID: \"9a48c702-25f2-4601-9257-8281d6735a35\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" Nov 03 14:32:46.374216 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:46.374183 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a48c702-25f2-4601-9257-8281d6735a35-workload-certs\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9\" (UID: \"9a48c702-25f2-4601-9257-8281d6735a35\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" Nov 03 14:32:46.374216 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:46.374202 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a48c702-25f2-4601-9257-8281d6735a35-workload-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9\" (UID: \"9a48c702-25f2-4601-9257-8281d6735a35\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" Nov 03 14:32:46.374331 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:46.374311 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a48c702-25f2-4601-9257-8281d6735a35-credential-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9\" (UID: \"9a48c702-25f2-4601-9257-8281d6735a35\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" Nov 03 14:32:46.374383 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:46.374324 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a48c702-25f2-4601-9257-8281d6735a35-istiod-ca-cert\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9\" (UID: \"9a48c702-25f2-4601-9257-8281d6735a35\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" Nov 03 14:32:46.376248 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:46.376219 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a48c702-25f2-4601-9257-8281d6735a35-istio-envoy\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9\" (UID: \"9a48c702-25f2-4601-9257-8281d6735a35\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" Nov 03 14:32:46.376576 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:46.376554 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a48c702-25f2-4601-9257-8281d6735a35-istio-podinfo\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9\" (UID: \"9a48c702-25f2-4601-9257-8281d6735a35\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" Nov 03 14:32:46.407975 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:46.407890 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a48c702-25f2-4601-9257-8281d6735a35-istio-token\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9\" (UID: \"9a48c702-25f2-4601-9257-8281d6735a35\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" Nov 03 14:32:46.408747 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:46.408722 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdg6g\" (UniqueName: \"kubernetes.io/projected/9a48c702-25f2-4601-9257-8281d6735a35-kube-api-access-vdg6g\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9\" (UID: \"9a48c702-25f2-4601-9257-8281d6735a35\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" Nov 03 14:32:46.486872 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:46.486760 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" Nov 03 14:32:46.818416 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:46.818106 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9"] Nov 03 14:32:46.818416 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:32:46.818166 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a48c702_25f2_4601_9257_8281d6735a35.slice/crio-6a83c679ab82c23a68f5b135cabb88c0edbae3ea1189b5692163f548d28fe70a WatchSource:0}: Error finding container 6a83c679ab82c23a68f5b135cabb88c0edbae3ea1189b5692163f548d28fe70a: Status 404 returned error can't find the container with id 6a83c679ab82c23a68f5b135cabb88c0edbae3ea1189b5692163f548d28fe70a Nov 03 14:32:46.826198 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:46.826154 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:32:46.826337 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:46.826249 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:32:46.826337 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:46.826292 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:32:47.868962 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:47.867696 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" event={"ID":"9a48c702-25f2-4601-9257-8281d6735a35","Type":"ContainerStarted","Data":"8b0afe4c6eefb518e1a65e06920e12673c31c0fa835cada1704cd43975dccb88"} Nov 03 14:32:47.868962 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:47.867772 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" event={"ID":"9a48c702-25f2-4601-9257-8281d6735a35","Type":"ContainerStarted","Data":"6a83c679ab82c23a68f5b135cabb88c0edbae3ea1189b5692163f548d28fe70a"} Nov 03 14:32:48.486906 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:48.486873 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" Nov 03 14:32:48.488965 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:48.488940 5060 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 14:32:48.489092 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:48.488994 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" podUID="9a48c702-25f2-4601-9257-8281d6735a35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 14:32:49.487981 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:49.487946 5060 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 14:32:49.488411 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:49.488011 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" podUID="9a48c702-25f2-4601-9257-8281d6735a35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 14:32:50.487732 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:50.487694 5060 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 14:32:50.487928 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:50.487761 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" podUID="9a48c702-25f2-4601-9257-8281d6735a35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 14:32:51.442470 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:51.442406 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" podStartSLOduration=5.442383436 podStartE2EDuration="5.442383436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:32:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:32:47.948451501 +0000 UTC m=+1301.837241084" watchObservedRunningTime="2026-11-03 14:32:51.442383436 +0000 UTC m=+1305.331173018" Nov 03 14:32:51.443592 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:51.443569 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-09d577dc-6c6fc7b44b-59zxt"] Nov 03 14:32:51.443964 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:51.443937 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d577dc-6c6fc7b44b-59zxt" podUID="34e15205-cba4-409a-abb1-1712d2292684" containerName="maas-api" containerID="cri-o://b5e4e668ece8b382954b91b8811cf1b0477d8eab7d0fedcf4404094365c29c35" gracePeriod=30 Nov 03 14:32:51.488873 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:51.487706 5060 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 14:32:51.488873 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:51.487765 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" podUID="9a48c702-25f2-4601-9257-8281d6735a35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 14:32:51.895330 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:51.895283 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d577dc-6c6fc7b44b-59zxt" Nov 03 14:32:51.969238 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:51.969202 5060 generic.go:356] "Generic (PLEG): container finished" podID="34e15205-cba4-409a-abb1-1712d2292684" containerID="b5e4e668ece8b382954b91b8811cf1b0477d8eab7d0fedcf4404094365c29c35" exitCode=0 Nov 03 14:32:51.969452 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:51.969413 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d577dc-6c6fc7b44b-59zxt" event={"ID":"34e15205-cba4-409a-abb1-1712d2292684","Type":"ContainerDied","Data":"b5e4e668ece8b382954b91b8811cf1b0477d8eab7d0fedcf4404094365c29c35"} Nov 03 14:32:51.969539 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:51.969503 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d577dc-6c6fc7b44b-59zxt" event={"ID":"34e15205-cba4-409a-abb1-1712d2292684","Type":"ContainerDied","Data":"69cf8a65820affcd09191b3317aacb317017956dcf88ea6908d71b1ef9720a92"} Nov 03 14:32:51.969596 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:51.969538 5060 scope.go:122] "RemoveContainer" containerID="b5e4e668ece8b382954b91b8811cf1b0477d8eab7d0fedcf4404094365c29c35" Nov 03 14:32:51.969704 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:51.969689 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d577dc-6c6fc7b44b-59zxt" Nov 03 14:32:51.983135 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:51.983105 5060 scope.go:122] "RemoveContainer" containerID="b5e4e668ece8b382954b91b8811cf1b0477d8eab7d0fedcf4404094365c29c35" Nov 03 14:32:51.983489 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:32:51.983468 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5e4e668ece8b382954b91b8811cf1b0477d8eab7d0fedcf4404094365c29c35\": container with ID starting with b5e4e668ece8b382954b91b8811cf1b0477d8eab7d0fedcf4404094365c29c35 not found: ID does not exist" containerID="b5e4e668ece8b382954b91b8811cf1b0477d8eab7d0fedcf4404094365c29c35" Nov 03 14:32:51.983578 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:51.983504 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5e4e668ece8b382954b91b8811cf1b0477d8eab7d0fedcf4404094365c29c35"} err="failed to get container status \"b5e4e668ece8b382954b91b8811cf1b0477d8eab7d0fedcf4404094365c29c35\": rpc error: code = NotFound desc = could not find container \"b5e4e668ece8b382954b91b8811cf1b0477d8eab7d0fedcf4404094365c29c35\": container with ID starting with b5e4e668ece8b382954b91b8811cf1b0477d8eab7d0fedcf4404094365c29c35 not found: ID does not exist" Nov 03 14:32:51.988887 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:51.988862 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34e15205-cba4-409a-abb1-1712d2292684-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34e15205-cba4-409a-abb1-1712d2292684-openshift-service-ca\") pod \"34e15205-cba4-409a-abb1-1712d2292684\" (UID: \"34e15205-cba4-409a-abb1-1712d2292684\") " Nov 03 14:32:51.989005 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:51.988914 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34e15205-cba4-409a-abb1-1712d2292684-metrics-certs\" (UniqueName: \"kubernetes.io/secret/34e15205-cba4-409a-abb1-1712d2292684-metrics-certs\") pod \"34e15205-cba4-409a-abb1-1712d2292684\" (UID: \"34e15205-cba4-409a-abb1-1712d2292684\") " Nov 03 14:32:51.989005 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:51.988952 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34e15205-cba4-409a-abb1-1712d2292684-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34e15205-cba4-409a-abb1-1712d2292684-maas-api-tls\") pod \"34e15205-cba4-409a-abb1-1712d2292684\" (UID: \"34e15205-cba4-409a-abb1-1712d2292684\") " Nov 03 14:32:51.989125 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:51.989062 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34e15205-cba4-409a-abb1-1712d2292684-kube-api-access-r2hn7\" (UniqueName: \"kubernetes.io/projected/34e15205-cba4-409a-abb1-1712d2292684-kube-api-access-r2hn7\") pod \"34e15205-cba4-409a-abb1-1712d2292684\" (UID: \"34e15205-cba4-409a-abb1-1712d2292684\") " Nov 03 14:32:51.990476 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:51.990437 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34e15205-cba4-409a-abb1-1712d2292684-openshift-service-ca" pod "34e15205-cba4-409a-abb1-1712d2292684" (UID: "34e15205-cba4-409a-abb1-1712d2292684"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:32:51.994724 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:51.994694 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34e15205-cba4-409a-abb1-1712d2292684-maas-api-tls" pod "34e15205-cba4-409a-abb1-1712d2292684" (UID: "34e15205-cba4-409a-abb1-1712d2292684"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:32:51.994957 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:51.994785 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34e15205-cba4-409a-abb1-1712d2292684-kube-api-access-r2hn7" pod "34e15205-cba4-409a-abb1-1712d2292684" (UID: "34e15205-cba4-409a-abb1-1712d2292684"). InnerVolumeSpecName "kube-api-access-r2hn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:32:51.996803 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:51.996779 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34e15205-cba4-409a-abb1-1712d2292684-metrics-certs" pod "34e15205-cba4-409a-abb1-1712d2292684" (UID: "34e15205-cba4-409a-abb1-1712d2292684"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:32:52.069893 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:52.063160 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-4a67e25f/e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p" podUID="0476cf91-73a3-4392-a1c7-1dc62e2a5b9f" containerName="main" containerID="cri-o://1f57cbde036137476a25b7f3998c5023bd9ba2846cf317aaf124e46a4362a312" gracePeriod=45 Nov 03 14:32:52.091220 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:52.091089 5060 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34e15205-cba4-409a-abb1-1712d2292684-openshift-service-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:32:52.091220 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:52.091130 5060 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34e15205-cba4-409a-abb1-1712d2292684-metrics-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:32:52.091220 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:52.091144 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34e15205-cba4-409a-abb1-1712d2292684-maas-api-tls\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:32:52.091220 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:52.091157 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2hn7\" (UniqueName: \"kubernetes.io/projected/34e15205-cba4-409a-abb1-1712d2292684-kube-api-access-r2hn7\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:32:52.344973 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:52.344871 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-09d577dc-6c6fc7b44b-59zxt"] Nov 03 14:32:52.351059 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:52.351028 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-09d577dc-6c6fc7b44b-59zxt"] Nov 03 14:32:52.390127 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:52.390101 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4a67e25f/e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p" Nov 03 14:32:52.487859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:52.487803 5060 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 14:32:52.488496 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:52.487883 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" podUID="9a48c702-25f2-4601-9257-8281d6735a35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 14:32:52.496247 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:52.496208 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f-tmp-dir\") pod \"0476cf91-73a3-4392-a1c7-1dc62e2a5b9f\" (UID: \"0476cf91-73a3-4392-a1c7-1dc62e2a5b9f\") " Nov 03 14:32:52.526860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:52.496917 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f-kube-api-access-997ct\" (UniqueName: \"kubernetes.io/projected/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f-kube-api-access-997ct\") pod \"0476cf91-73a3-4392-a1c7-1dc62e2a5b9f\" (UID: \"0476cf91-73a3-4392-a1c7-1dc62e2a5b9f\") " Nov 03 14:32:52.532346 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:52.532307 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f-dshm\") pod \"0476cf91-73a3-4392-a1c7-1dc62e2a5b9f\" (UID: \"0476cf91-73a3-4392-a1c7-1dc62e2a5b9f\") " Nov 03 14:32:52.532626 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:52.532603 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f-tls-certs\" (UniqueName: \"kubernetes.io/secret/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f-tls-certs\") pod \"0476cf91-73a3-4392-a1c7-1dc62e2a5b9f\" (UID: \"0476cf91-73a3-4392-a1c7-1dc62e2a5b9f\") " Nov 03 14:32:52.532776 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:52.532758 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f-model-cache\") pod \"0476cf91-73a3-4392-a1c7-1dc62e2a5b9f\" (UID: \"0476cf91-73a3-4392-a1c7-1dc62e2a5b9f\") " Nov 03 14:32:52.533057 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:52.533037 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f-home\" (UniqueName: \"kubernetes.io/empty-dir/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f-home\") pod \"0476cf91-73a3-4392-a1c7-1dc62e2a5b9f\" (UID: \"0476cf91-73a3-4392-a1c7-1dc62e2a5b9f\") " Nov 03 14:32:52.534059 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:52.496728 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f-tmp-dir" pod "0476cf91-73a3-4392-a1c7-1dc62e2a5b9f" (UID: "0476cf91-73a3-4392-a1c7-1dc62e2a5b9f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:32:52.534197 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:52.534034 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f-home" pod "0476cf91-73a3-4392-a1c7-1dc62e2a5b9f" (UID: "0476cf91-73a3-4392-a1c7-1dc62e2a5b9f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:32:52.536200 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:52.536162 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f-model-cache" pod "0476cf91-73a3-4392-a1c7-1dc62e2a5b9f" (UID: "0476cf91-73a3-4392-a1c7-1dc62e2a5b9f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:32:52.538346 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:52.538315 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f-kube-api-access-997ct" pod "0476cf91-73a3-4392-a1c7-1dc62e2a5b9f" (UID: "0476cf91-73a3-4392-a1c7-1dc62e2a5b9f"). InnerVolumeSpecName "kube-api-access-997ct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:32:52.541607 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:52.541584 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f-dshm" pod "0476cf91-73a3-4392-a1c7-1dc62e2a5b9f" (UID: "0476cf91-73a3-4392-a1c7-1dc62e2a5b9f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:32:52.549333 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:52.549296 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f-tls-certs" pod "0476cf91-73a3-4392-a1c7-1dc62e2a5b9f" (UID: "0476cf91-73a3-4392-a1c7-1dc62e2a5b9f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:32:52.645862 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:52.639944 5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f-home\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:32:52.645862 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:52.639988 5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f-tmp-dir\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:32:52.645862 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:52.640010 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-997ct\" (UniqueName: \"kubernetes.io/projected/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f-kube-api-access-997ct\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:32:52.645862 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:52.640023 5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f-dshm\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:32:52.645862 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:52.640036 5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f-tls-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:32:52.645862 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:52.640050 5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f-model-cache\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:32:52.977932 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:52.977898 5060 generic.go:356] "Generic (PLEG): container finished" podID="0476cf91-73a3-4392-a1c7-1dc62e2a5b9f" containerID="1f57cbde036137476a25b7f3998c5023bd9ba2846cf317aaf124e46a4362a312" exitCode=0 Nov 03 14:32:52.978134 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:52.977939 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4a67e25f/e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p" Nov 03 14:32:52.978134 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:52.977978 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4a67e25f/e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p" event={"ID":"0476cf91-73a3-4392-a1c7-1dc62e2a5b9f","Type":"ContainerDied","Data":"1f57cbde036137476a25b7f3998c5023bd9ba2846cf317aaf124e46a4362a312"} Nov 03 14:32:52.978134 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:52.978025 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4a67e25f/e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p" event={"ID":"0476cf91-73a3-4392-a1c7-1dc62e2a5b9f","Type":"ContainerDied","Data":"bb3668af23722e77664db87101b4567deca610001e8563fad5894fa09f57e145"} Nov 03 14:32:52.978134 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:52.978050 5060 scope.go:122] "RemoveContainer" containerID="1f57cbde036137476a25b7f3998c5023bd9ba2846cf317aaf124e46a4362a312" Nov 03 14:32:52.988002 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:52.987982 5060 scope.go:122] "RemoveContainer" containerID="1f57cbde036137476a25b7f3998c5023bd9ba2846cf317aaf124e46a4362a312" Nov 03 14:32:52.988243 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:32:52.988226 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f57cbde036137476a25b7f3998c5023bd9ba2846cf317aaf124e46a4362a312\": container with ID starting with 1f57cbde036137476a25b7f3998c5023bd9ba2846cf317aaf124e46a4362a312 not found: ID does not exist" containerID="1f57cbde036137476a25b7f3998c5023bd9ba2846cf317aaf124e46a4362a312" Nov 03 14:32:52.988302 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:52.988262 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f57cbde036137476a25b7f3998c5023bd9ba2846cf317aaf124e46a4362a312"} err="failed to get container status \"1f57cbde036137476a25b7f3998c5023bd9ba2846cf317aaf124e46a4362a312\": rpc error: code = NotFound desc = could not find container \"1f57cbde036137476a25b7f3998c5023bd9ba2846cf317aaf124e46a4362a312\": container with ID starting with 1f57cbde036137476a25b7f3998c5023bd9ba2846cf317aaf124e46a4362a312 not found: ID does not exist" Nov 03 14:32:53.044081 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:53.044040 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-4a67e25f/e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p"] Nov 03 14:32:53.053494 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:53.053462 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-4a67e25f/e2e-lifecycle-model-4a67e25f-kserve-556b798877-z8h8p"] Nov 03 14:32:53.487447 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:53.487409 5060 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 14:32:53.487640 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:53.487507 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" podUID="9a48c702-25f2-4601-9257-8281d6735a35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 14:32:54.224859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.224639 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j"] Nov 03 14:32:54.231856 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.225545 5060 cpu_manager.go:414] "containerMap: removing container" podUID="34e15205-cba4-409a-abb1-1712d2292684" containerName="maas-api" Nov 03 14:32:54.231856 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.225567 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34e15205-cba4-409a-abb1-1712d2292684" containerName="maas-api" Nov 03 14:32:54.231856 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.225598 5060 cpu_manager.go:414] "containerMap: removing container" podUID="0476cf91-73a3-4392-a1c7-1dc62e2a5b9f" containerName="main" Nov 03 14:32:54.231856 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.225606 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0476cf91-73a3-4392-a1c7-1dc62e2a5b9f" containerName="main" Nov 03 14:32:54.231856 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.225753 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="34e15205-cba4-409a-abb1-1712d2292684" containerName="maas-api" Nov 03 14:32:54.231856 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.225770 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="0476cf91-73a3-4392-a1c7-1dc62e2a5b9f" containerName="main" Nov 03 14:32:54.256743 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.256280 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" Nov 03 14:32:54.275464 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.275413 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-95af53a1-openshift-default-dockercfg-xdgqc\"" Nov 03 14:32:54.275645 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.275558 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0476cf91-73a3-4392-a1c7-1dc62e2a5b9f" path="/var/lib/kubelet/pods/0476cf91-73a3-4392-a1c7-1dc62e2a5b9f/volumes" Nov 03 14:32:54.276426 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.276398 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34e15205-cba4-409a-abb1-1712d2292684" path="/var/lib/kubelet/pods/34e15205-cba4-409a-abb1-1712d2292684/volumes" Nov 03 14:32:54.277076 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.277057 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j"] Nov 03 14:32:54.371952 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.371914 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/599e862d-75d4-40f6-9750-806110c5d308-workload-socket\") pod \"e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j\" (UID: \"599e862d-75d4-40f6-9750-806110c5d308\") " pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" Nov 03 14:32:54.371952 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.371953 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/599e862d-75d4-40f6-9750-806110c5d308-istio-podinfo\") pod \"e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j\" (UID: \"599e862d-75d4-40f6-9750-806110c5d308\") " pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" Nov 03 14:32:54.372210 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.372133 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/599e862d-75d4-40f6-9750-806110c5d308-credential-socket\") pod \"e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j\" (UID: \"599e862d-75d4-40f6-9750-806110c5d308\") " pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" Nov 03 14:32:54.372210 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.372183 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/599e862d-75d4-40f6-9750-806110c5d308-istio-envoy\") pod \"e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j\" (UID: \"599e862d-75d4-40f6-9750-806110c5d308\") " pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" Nov 03 14:32:54.372358 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.372322 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/599e862d-75d4-40f6-9750-806110c5d308-istio-token\") pod \"e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j\" (UID: \"599e862d-75d4-40f6-9750-806110c5d308\") " pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" Nov 03 14:32:54.372503 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.372369 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/599e862d-75d4-40f6-9750-806110c5d308-workload-certs\") pod \"e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j\" (UID: \"599e862d-75d4-40f6-9750-806110c5d308\") " pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" Nov 03 14:32:54.372503 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.372407 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n7sl\" (UniqueName: \"kubernetes.io/projected/599e862d-75d4-40f6-9750-806110c5d308-kube-api-access-8n7sl\") pod \"e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j\" (UID: \"599e862d-75d4-40f6-9750-806110c5d308\") " pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" Nov 03 14:32:54.372613 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.372577 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/599e862d-75d4-40f6-9750-806110c5d308-istiod-ca-cert\") pod \"e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j\" (UID: \"599e862d-75d4-40f6-9750-806110c5d308\") " pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" Nov 03 14:32:54.372677 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.372658 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/599e862d-75d4-40f6-9750-806110c5d308-istio-data\") pod \"e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j\" (UID: \"599e862d-75d4-40f6-9750-806110c5d308\") " pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" Nov 03 14:32:54.473789 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.473744 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/599e862d-75d4-40f6-9750-806110c5d308-istiod-ca-cert\") pod \"e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j\" (UID: \"599e862d-75d4-40f6-9750-806110c5d308\") " pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" Nov 03 14:32:54.474131 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.474103 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/599e862d-75d4-40f6-9750-806110c5d308-istio-data\") pod \"e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j\" (UID: \"599e862d-75d4-40f6-9750-806110c5d308\") " pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" Nov 03 14:32:54.474310 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.474293 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/599e862d-75d4-40f6-9750-806110c5d308-workload-socket\") pod \"e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j\" (UID: \"599e862d-75d4-40f6-9750-806110c5d308\") " pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" Nov 03 14:32:54.474418 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.474404 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/599e862d-75d4-40f6-9750-806110c5d308-istio-podinfo\") pod \"e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j\" (UID: \"599e862d-75d4-40f6-9750-806110c5d308\") " pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" Nov 03 14:32:54.474586 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.474569 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/599e862d-75d4-40f6-9750-806110c5d308-credential-socket\") pod \"e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j\" (UID: \"599e862d-75d4-40f6-9750-806110c5d308\") " pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" Nov 03 14:32:54.474706 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.474690 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/599e862d-75d4-40f6-9750-806110c5d308-istio-envoy\") pod \"e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j\" (UID: \"599e862d-75d4-40f6-9750-806110c5d308\") " pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" Nov 03 14:32:54.474905 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.474860 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/599e862d-75d4-40f6-9750-806110c5d308-istio-token\") pod \"e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j\" (UID: \"599e862d-75d4-40f6-9750-806110c5d308\") " pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" Nov 03 14:32:54.475042 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.475026 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/599e862d-75d4-40f6-9750-806110c5d308-workload-certs\") pod \"e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j\" (UID: \"599e862d-75d4-40f6-9750-806110c5d308\") " pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" Nov 03 14:32:54.475166 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.475141 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/599e862d-75d4-40f6-9750-806110c5d308-istio-data\") pod \"e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j\" (UID: \"599e862d-75d4-40f6-9750-806110c5d308\") " pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" Nov 03 14:32:54.475262 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.475161 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n7sl\" (UniqueName: \"kubernetes.io/projected/599e862d-75d4-40f6-9750-806110c5d308-kube-api-access-8n7sl\") pod \"e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j\" (UID: \"599e862d-75d4-40f6-9750-806110c5d308\") " pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" Nov 03 14:32:54.483544 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.474931 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/599e862d-75d4-40f6-9750-806110c5d308-workload-socket\") pod \"e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j\" (UID: \"599e862d-75d4-40f6-9750-806110c5d308\") " pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" Nov 03 14:32:54.483544 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.476079 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/599e862d-75d4-40f6-9750-806110c5d308-workload-certs\") pod \"e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j\" (UID: \"599e862d-75d4-40f6-9750-806110c5d308\") " pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" Nov 03 14:32:54.483544 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.474646 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/599e862d-75d4-40f6-9750-806110c5d308-istiod-ca-cert\") pod \"e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j\" (UID: \"599e862d-75d4-40f6-9750-806110c5d308\") " pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" Nov 03 14:32:54.545856 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.545115 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/599e862d-75d4-40f6-9750-806110c5d308-credential-socket\") pod \"e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j\" (UID: \"599e862d-75d4-40f6-9750-806110c5d308\") " pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" Nov 03 14:32:54.546051 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.545905 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/599e862d-75d4-40f6-9750-806110c5d308-istio-envoy\") pod \"e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j\" (UID: \"599e862d-75d4-40f6-9750-806110c5d308\") " pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" Nov 03 14:32:54.564982 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.546548 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/599e862d-75d4-40f6-9750-806110c5d308-istio-podinfo\") pod \"e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j\" (UID: \"599e862d-75d4-40f6-9750-806110c5d308\") " pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" Nov 03 14:32:54.569981 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.569946 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n7sl\" (UniqueName: \"kubernetes.io/projected/599e862d-75d4-40f6-9750-806110c5d308-kube-api-access-8n7sl\") pod \"e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j\" (UID: \"599e862d-75d4-40f6-9750-806110c5d308\") " pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" Nov 03 14:32:54.581622 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.581588 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/599e862d-75d4-40f6-9750-806110c5d308-istio-token\") pod \"e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j\" (UID: \"599e862d-75d4-40f6-9750-806110c5d308\") " pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" Nov 03 14:32:54.603968 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.603938 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" Nov 03 14:32:54.604987 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.604729 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" Nov 03 14:32:54.605978 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.605921 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" Nov 03 14:32:54.882466 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:54.882432 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" Nov 03 14:32:55.234713 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:55.234631 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j"] Nov 03 14:32:55.238868 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:32:55.238824 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod599e862d_75d4_40f6_9750_806110c5d308.slice/crio-77ffade70e8d636ee736a10d16798ea286568c98f8d3ce09d594f40e2961f00c WatchSource:0}: Error finding container 77ffade70e8d636ee736a10d16798ea286568c98f8d3ce09d594f40e2961f00c: Status 404 returned error can't find the container with id 77ffade70e8d636ee736a10d16798ea286568c98f8d3ce09d594f40e2961f00c Nov 03 14:32:55.241366 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:55.241334 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:32:55.241462 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:55.241399 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:32:55.241462 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:55.241428 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:32:56.009747 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:56.009707 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" event={"ID":"599e862d-75d4-40f6-9750-806110c5d308","Type":"ContainerStarted","Data":"eaebb2c149f405314e678a3dae00f4591fcef556ffa1d750490a75466f5f498e"} Nov 03 14:32:56.010001 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:56.009764 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" event={"ID":"599e862d-75d4-40f6-9750-806110c5d308","Type":"ContainerStarted","Data":"77ffade70e8d636ee736a10d16798ea286568c98f8d3ce09d594f40e2961f00c"} Nov 03 14:32:56.073705 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:56.073650 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" podStartSLOduration=2.073635339 podStartE2EDuration="2.073635339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:32:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:32:56.071810497 +0000 UTC m=+1309.960600078" watchObservedRunningTime="2026-11-03 14:32:56.073635339 +0000 UTC m=+1309.962424920" Nov 03 14:32:56.883014 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:56.882977 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" Nov 03 14:32:56.888457 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:56.888430 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" Nov 03 14:32:57.016966 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:57.016932 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" Nov 03 14:32:57.017744 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:32:57.017716 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" Nov 03 14:33:07.261134 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:07.261104 5060 scope.go:122] "RemoveContainer" containerID="343bc6ba378af87be42a14a083e8fd195974426ab4ad1d1fa7e75913b795bf8e" Nov 03 14:33:07.271178 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:07.271152 5060 scope.go:122] "RemoveContainer" containerID="9ea5a9be6931d78841bd7f963c782b594996607eff6ed6f65a70532210c29779" Nov 03 14:33:07.280856 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:07.280825 5060 scope.go:122] "RemoveContainer" containerID="80b31063a03c108d34d4a251b774e1cf03d4a8e58f562a1dce5f256007e3b14c" Nov 03 14:33:07.290208 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:07.290188 5060 scope.go:122] "RemoveContainer" containerID="a879e953cb0939d131a77c634d70055dd83ac077134207efb3091b911f88bf05" Nov 03 14:33:09.441897 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:33:09.441852 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:33:16.548325 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:16.548290 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-95af53a1-bd758ddff-zngcj"] Nov 03 14:33:16.554232 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:16.554204 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-95af53a1-bd758ddff-zngcj" Nov 03 14:33:16.592792 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:16.592041 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-95af53a1-serving-cert\"" Nov 03 14:33:16.596534 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:16.595952 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-95af53a1-bd758ddff-zngcj"] Nov 03 14:33:16.750899 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:16.727941 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/957440c5-6bdc-42eb-ab1e-9e78e026ca43-metrics-certs\") pod \"maas-api-e2e-derive-95af53a1-bd758ddff-zngcj\" (UID: \"957440c5-6bdc-42eb-ab1e-9e78e026ca43\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-95af53a1-bd758ddff-zngcj" Nov 03 14:33:16.751221 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:16.751192 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzwsw\" (UniqueName: \"kubernetes.io/projected/957440c5-6bdc-42eb-ab1e-9e78e026ca43-kube-api-access-lzwsw\") pod \"maas-api-e2e-derive-95af53a1-bd758ddff-zngcj\" (UID: \"957440c5-6bdc-42eb-ab1e-9e78e026ca43\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-95af53a1-bd758ddff-zngcj" Nov 03 14:33:16.751528 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:16.751504 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/957440c5-6bdc-42eb-ab1e-9e78e026ca43-maas-api-tls\") pod \"maas-api-e2e-derive-95af53a1-bd758ddff-zngcj\" (UID: \"957440c5-6bdc-42eb-ab1e-9e78e026ca43\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-95af53a1-bd758ddff-zngcj" Nov 03 14:33:16.751772 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:16.751757 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/957440c5-6bdc-42eb-ab1e-9e78e026ca43-openshift-service-ca\") pod \"maas-api-e2e-derive-95af53a1-bd758ddff-zngcj\" (UID: \"957440c5-6bdc-42eb-ab1e-9e78e026ca43\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-95af53a1-bd758ddff-zngcj" Nov 03 14:33:16.853802 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:16.852970 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzwsw\" (UniqueName: \"kubernetes.io/projected/957440c5-6bdc-42eb-ab1e-9e78e026ca43-kube-api-access-lzwsw\") pod \"maas-api-e2e-derive-95af53a1-bd758ddff-zngcj\" (UID: \"957440c5-6bdc-42eb-ab1e-9e78e026ca43\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-95af53a1-bd758ddff-zngcj" Nov 03 14:33:16.853802 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:16.853027 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/957440c5-6bdc-42eb-ab1e-9e78e026ca43-maas-api-tls\") pod \"maas-api-e2e-derive-95af53a1-bd758ddff-zngcj\" (UID: \"957440c5-6bdc-42eb-ab1e-9e78e026ca43\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-95af53a1-bd758ddff-zngcj" Nov 03 14:33:16.853802 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:16.853143 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/957440c5-6bdc-42eb-ab1e-9e78e026ca43-openshift-service-ca\") pod \"maas-api-e2e-derive-95af53a1-bd758ddff-zngcj\" (UID: \"957440c5-6bdc-42eb-ab1e-9e78e026ca43\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-95af53a1-bd758ddff-zngcj" Nov 03 14:33:16.853802 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:16.853314 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/957440c5-6bdc-42eb-ab1e-9e78e026ca43-metrics-certs\") pod \"maas-api-e2e-derive-95af53a1-bd758ddff-zngcj\" (UID: \"957440c5-6bdc-42eb-ab1e-9e78e026ca43\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-95af53a1-bd758ddff-zngcj" Nov 03 14:33:16.858095 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:33:16.854023 5060 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-95af53a1-serving-cert: secret "maas-api-e2e-derive-95af53a1-serving-cert" not found Nov 03 14:33:16.858095 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:33:16.854099 5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/957440c5-6bdc-42eb-ab1e-9e78e026ca43-maas-api-tls podName:957440c5-6bdc-42eb-ab1e-9e78e026ca43 nodeName:}" failed. No retries permitted until 2026-11-03 14:33:17.354076433 +0000 UTC m=+1331.242866010 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/957440c5-6bdc-42eb-ab1e-9e78e026ca43-maas-api-tls") pod "maas-api-e2e-derive-95af53a1-bd758ddff-zngcj" (UID: "957440c5-6bdc-42eb-ab1e-9e78e026ca43") : secret "maas-api-e2e-derive-95af53a1-serving-cert" not found Nov 03 14:33:16.858095 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:16.855238 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/957440c5-6bdc-42eb-ab1e-9e78e026ca43-openshift-service-ca\") pod \"maas-api-e2e-derive-95af53a1-bd758ddff-zngcj\" (UID: \"957440c5-6bdc-42eb-ab1e-9e78e026ca43\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-95af53a1-bd758ddff-zngcj" Nov 03 14:33:16.864241 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:16.863661 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/957440c5-6bdc-42eb-ab1e-9e78e026ca43-metrics-certs\") pod \"maas-api-e2e-derive-95af53a1-bd758ddff-zngcj\" (UID: \"957440c5-6bdc-42eb-ab1e-9e78e026ca43\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-95af53a1-bd758ddff-zngcj" Nov 03 14:33:16.876500 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:16.876477 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzwsw\" (UniqueName: \"kubernetes.io/projected/957440c5-6bdc-42eb-ab1e-9e78e026ca43-kube-api-access-lzwsw\") pod \"maas-api-e2e-derive-95af53a1-bd758ddff-zngcj\" (UID: \"957440c5-6bdc-42eb-ab1e-9e78e026ca43\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-95af53a1-bd758ddff-zngcj" Nov 03 14:33:17.359033 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:17.358993 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/957440c5-6bdc-42eb-ab1e-9e78e026ca43-maas-api-tls\") pod \"maas-api-e2e-derive-95af53a1-bd758ddff-zngcj\" (UID: \"957440c5-6bdc-42eb-ab1e-9e78e026ca43\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-95af53a1-bd758ddff-zngcj" Nov 03 14:33:17.361368 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:17.361342 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/957440c5-6bdc-42eb-ab1e-9e78e026ca43-maas-api-tls\") pod \"maas-api-e2e-derive-95af53a1-bd758ddff-zngcj\" (UID: \"957440c5-6bdc-42eb-ab1e-9e78e026ca43\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-95af53a1-bd758ddff-zngcj" Nov 03 14:33:17.473939 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:17.473908 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-95af53a1-bd758ddff-zngcj" Nov 03 14:33:17.780023 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:17.779993 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-95af53a1-bd758ddff-zngcj"] Nov 03 14:33:17.781129 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:33:17.781102 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod957440c5_6bdc_42eb_ab1e_9e78e026ca43.slice/crio-026ae8cc816ea9be8070f90ed54d0a1f353a3e897fe9413f3973594987869b55 WatchSource:0}: Error finding container 026ae8cc816ea9be8070f90ed54d0a1f353a3e897fe9413f3973594987869b55: Status 404 returned error can't find the container with id 026ae8cc816ea9be8070f90ed54d0a1f353a3e897fe9413f3973594987869b55 Nov 03 14:33:18.232463 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:18.232413 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-95af53a1-bd758ddff-zngcj" event={"ID":"957440c5-6bdc-42eb-ab1e-9e78e026ca43","Type":"ContainerStarted","Data":"bd53ee25ffc52cdb9b52d7f04647ba33e9dae64e4dfdd27c24f023c4eaf1da5e"} Nov 03 14:33:18.232662 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:18.232482 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-95af53a1-bd758ddff-zngcj" event={"ID":"957440c5-6bdc-42eb-ab1e-9e78e026ca43","Type":"ContainerStarted","Data":"026ae8cc816ea9be8070f90ed54d0a1f353a3e897fe9413f3973594987869b55"} Nov 03 14:33:18.232734 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:18.232686 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-95af53a1-bd758ddff-zngcj" Nov 03 14:33:18.282107 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:18.282048 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-95af53a1-bd758ddff-zngcj" podStartSLOduration=1.920068347 podStartE2EDuration="2.282035469s" totalImagesPullingTime="361.967122ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:33:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:33:18.278372709 +0000 UTC m=+1332.167162300" watchObservedRunningTime="2026-11-03 14:33:18.282035469 +0000 UTC m=+1332.170825049" Nov 03 14:33:24.249600 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:24.249574 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-95af53a1-bd758ddff-zngcj" Nov 03 14:33:26.044159 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:26.044124 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-655b301b-brlf2"] Nov 03 14:33:26.057077 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:26.057049 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-655b301b-brlf2"] Nov 03 14:33:26.251731 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:26.251699 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1" path="/var/lib/kubelet/pods/af5d2cef-69d9-4ed2-a46c-b4d84f40d7e1/volumes" Nov 03 14:33:27.071456 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:27.071424 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p"] Nov 03 14:33:27.072176 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:27.072142 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p" podUID="393199fc-cbb3-4681-a6ee-cce53def4689" containerName="maas-api" containerID="cri-o://ca828d6ecf85ab2bfd1ea064cfafe4bd2ba5909fa0de60b6432c2a4af19a5faf" gracePeriod=30 Nov 03 14:33:27.306054 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:27.305785 5060 generic.go:356] "Generic (PLEG): container finished" podID="393199fc-cbb3-4681-a6ee-cce53def4689" containerID="ca828d6ecf85ab2bfd1ea064cfafe4bd2ba5909fa0de60b6432c2a4af19a5faf" exitCode=0 Nov 03 14:33:27.306054 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:27.305987 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p" event={"ID":"393199fc-cbb3-4681-a6ee-cce53def4689","Type":"ContainerDied","Data":"ca828d6ecf85ab2bfd1ea064cfafe4bd2ba5909fa0de60b6432c2a4af19a5faf"} Nov 03 14:33:27.440770 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:27.440453 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p" Nov 03 14:33:27.493073 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:27.493039 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/393199fc-cbb3-4681-a6ee-cce53def4689-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/393199fc-cbb3-4681-a6ee-cce53def4689-openshift-service-ca\") pod \"393199fc-cbb3-4681-a6ee-cce53def4689\" (UID: \"393199fc-cbb3-4681-a6ee-cce53def4689\") " Nov 03 14:33:27.493273 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:27.493116 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/393199fc-cbb3-4681-a6ee-cce53def4689-metrics-certs\" (UniqueName: \"kubernetes.io/secret/393199fc-cbb3-4681-a6ee-cce53def4689-metrics-certs\") pod \"393199fc-cbb3-4681-a6ee-cce53def4689\" (UID: \"393199fc-cbb3-4681-a6ee-cce53def4689\") " Nov 03 14:33:27.493273 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:27.493217 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/393199fc-cbb3-4681-a6ee-cce53def4689-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/393199fc-cbb3-4681-a6ee-cce53def4689-maas-api-tls\") pod \"393199fc-cbb3-4681-a6ee-cce53def4689\" (UID: \"393199fc-cbb3-4681-a6ee-cce53def4689\") " Nov 03 14:33:27.493399 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:27.493317 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/393199fc-cbb3-4681-a6ee-cce53def4689-kube-api-access-d9k8p\" (UniqueName: \"kubernetes.io/projected/393199fc-cbb3-4681-a6ee-cce53def4689-kube-api-access-d9k8p\") pod \"393199fc-cbb3-4681-a6ee-cce53def4689\" (UID: \"393199fc-cbb3-4681-a6ee-cce53def4689\") " Nov 03 14:33:27.493501 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:27.493467 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/393199fc-cbb3-4681-a6ee-cce53def4689-openshift-service-ca" pod "393199fc-cbb3-4681-a6ee-cce53def4689" (UID: "393199fc-cbb3-4681-a6ee-cce53def4689"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:33:27.493810 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:27.493783 5060 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/393199fc-cbb3-4681-a6ee-cce53def4689-openshift-service-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:33:27.506863 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:27.496326 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/393199fc-cbb3-4681-a6ee-cce53def4689-kube-api-access-d9k8p" pod "393199fc-cbb3-4681-a6ee-cce53def4689" (UID: "393199fc-cbb3-4681-a6ee-cce53def4689"). InnerVolumeSpecName "kube-api-access-d9k8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:33:27.506863 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:27.496336 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/393199fc-cbb3-4681-a6ee-cce53def4689-maas-api-tls" pod "393199fc-cbb3-4681-a6ee-cce53def4689" (UID: "393199fc-cbb3-4681-a6ee-cce53def4689"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:33:27.506863 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:27.496381 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/393199fc-cbb3-4681-a6ee-cce53def4689-metrics-certs" pod "393199fc-cbb3-4681-a6ee-cce53def4689" (UID: "393199fc-cbb3-4681-a6ee-cce53def4689"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:33:27.595595 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:27.595557 5060 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/393199fc-cbb3-4681-a6ee-cce53def4689-metrics-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:33:27.595595 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:27.595594 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/393199fc-cbb3-4681-a6ee-cce53def4689-maas-api-tls\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:33:27.595793 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:27.595609 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9k8p\" (UniqueName: \"kubernetes.io/projected/393199fc-cbb3-4681-a6ee-cce53def4689-kube-api-access-d9k8p\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:33:28.314326 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:28.314299 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p" Nov 03 14:33:28.314728 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:28.314296 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p" event={"ID":"393199fc-cbb3-4681-a6ee-cce53def4689","Type":"ContainerDied","Data":"1da03098efefbe5bc81458ba4d9cffdaea71f210cf841d738a7edafa784e68c9"} Nov 03 14:33:28.314728 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:28.314451 5060 scope.go:122] "RemoveContainer" containerID="ca828d6ecf85ab2bfd1ea064cfafe4bd2ba5909fa0de60b6432c2a4af19a5faf" Nov 03 14:33:28.375170 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:28.375117 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p"] Nov 03 14:33:28.386489 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:28.386458 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4a67e25f-57fd484774-9jn7p"] Nov 03 14:33:30.258023 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:30.257975 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="393199fc-cbb3-4681-a6ee-cce53def4689" path="/var/lib/kubelet/pods/393199fc-cbb3-4681-a6ee-cce53def4689/volumes" Nov 03 14:33:31.333184 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:31.333150 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs"] Nov 03 14:33:31.334084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:31.334064 5060 cpu_manager.go:414] "containerMap: removing container" podUID="393199fc-cbb3-4681-a6ee-cce53def4689" containerName="maas-api" Nov 03 14:33:31.334162 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:31.334088 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="393199fc-cbb3-4681-a6ee-cce53def4689" containerName="maas-api" Nov 03 14:33:31.334301 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:31.334281 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="393199fc-cbb3-4681-a6ee-cce53def4689" containerName="maas-api" Nov 03 14:33:31.339610 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:31.339587 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" Nov 03 14:33:31.359826 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:31.359800 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs"] Nov 03 14:33:31.537475 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:31.537443 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-workload-certs\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs\" (UID: \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" Nov 03 14:33:31.537644 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:31.537489 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-workload-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs\" (UID: \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" Nov 03 14:33:31.537644 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:31.537585 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-istio-token\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs\" (UID: \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" Nov 03 14:33:31.537712 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:31.537697 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-istio-data\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs\" (UID: \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" Nov 03 14:33:31.537743 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:31.537728 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-istio-podinfo\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs\" (UID: \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" Nov 03 14:33:31.537829 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:31.537815 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-credential-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs\" (UID: \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" Nov 03 14:33:31.538001 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:31.537906 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-istio-envoy\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs\" (UID: \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" Nov 03 14:33:31.538001 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:31.537936 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-istiod-ca-cert\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs\" (UID: \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" Nov 03 14:33:31.538001 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:31.537952 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlcjp\" (UniqueName: \"kubernetes.io/projected/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-kube-api-access-xlcjp\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs\" (UID: \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" Nov 03 14:33:31.639447 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:31.639360 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-istio-token\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs\" (UID: \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" Nov 03 14:33:31.639616 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:31.639492 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-istio-data\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs\" (UID: \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" Nov 03 14:33:31.639616 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:31.639519 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-istio-podinfo\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs\" (UID: \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" Nov 03 14:33:31.639616 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:31.639604 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-credential-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs\" (UID: \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" Nov 03 14:33:31.639789 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:31.639657 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-istio-envoy\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs\" (UID: \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" Nov 03 14:33:31.639789 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:31.639700 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-istiod-ca-cert\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs\" (UID: \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" Nov 03 14:33:31.639789 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:31.639727 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlcjp\" (UniqueName: \"kubernetes.io/projected/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-kube-api-access-xlcjp\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs\" (UID: \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" Nov 03 14:33:31.639976 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:31.639801 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-workload-certs\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs\" (UID: \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" Nov 03 14:33:31.639976 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:31.639865 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-workload-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs\" (UID: \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" Nov 03 14:33:31.640083 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:31.640017 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-istio-data\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs\" (UID: \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" Nov 03 14:33:31.640318 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:31.640294 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-workload-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs\" (UID: \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" Nov 03 14:33:31.640487 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:31.640350 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-credential-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs\" (UID: \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" Nov 03 14:33:31.640597 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:31.640532 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-workload-certs\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs\" (UID: \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" Nov 03 14:33:31.640719 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:31.640700 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-istiod-ca-cert\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs\" (UID: \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" Nov 03 14:33:31.642954 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:31.642930 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-istio-podinfo\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs\" (UID: \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" Nov 03 14:33:31.646114 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:31.646091 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-istio-envoy\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs\" (UID: \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" Nov 03 14:33:31.662933 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:31.662907 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-istio-token\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs\" (UID: \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" Nov 03 14:33:31.663064 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:31.663049 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlcjp\" (UniqueName: \"kubernetes.io/projected/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-kube-api-access-xlcjp\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs\" (UID: \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" Nov 03 14:33:31.956472 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:31.956383 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" Nov 03 14:33:32.338655 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:32.338598 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs"] Nov 03 14:33:32.342468 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:32.342415 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:33:32.342570 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:32.342540 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:33:32.342611 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:32.342580 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:33:32.384168 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:32.384138 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" event={"ID":"ae832eba-b22b-41c2-8b30-a4e5d7d3afac","Type":"ContainerStarted","Data":"9109e1d75e0db74e060ef4c990b507ca43624139c63c7c20fbe95bfa21c137f8"} Nov 03 14:33:33.393040 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:33.393000 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" event={"ID":"ae832eba-b22b-41c2-8b30-a4e5d7d3afac","Type":"ContainerStarted","Data":"6850cc1b011df07b5b361cdb6d70008716799479a05dabfe2f459d76b92a8d85"} Nov 03 14:33:33.473602 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:33.473550 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" podStartSLOduration=2.473535405 podStartE2EDuration="2.473535405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:33:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:33:33.459164367 +0000 UTC m=+1347.347953949" watchObservedRunningTime="2026-11-03 14:33:33.473535405 +0000 UTC m=+1347.362324985" Nov 03 14:33:33.959677 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:33.958173 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" Nov 03 14:33:33.959677 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:33.959643 5060 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 14:33:33.959995 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:33.959697 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" podUID="ae832eba-b22b-41c2-8b30-a4e5d7d3afac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 14:33:34.084859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:34.083856 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj"] Nov 03 14:33:34.084859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:34.084113 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" podUID="07fa990f-370e-4d84-b7e6-340f4caa6348" containerName="istio-proxy" containerID="cri-o://73f3789700b9b3481bf161f6f8c3a04b171b4965563257f488c66bcdbe9f5452" gracePeriod=30 Nov 03 14:33:34.959859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:34.959704 5060 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 14:33:34.959859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:34.959771 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" podUID="ae832eba-b22b-41c2-8b30-a4e5d7d3afac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 14:33:35.957374 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:35.957332 5060 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 14:33:35.957569 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:35.957391 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" podUID="ae832eba-b22b-41c2-8b30-a4e5d7d3afac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 14:33:36.958738 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:36.958695 5060 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 14:33:36.959803 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:36.958766 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" podUID="ae832eba-b22b-41c2-8b30-a4e5d7d3afac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 14:33:37.957114 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:37.957074 5060 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 14:33:37.957359 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:37.957135 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" podUID="ae832eba-b22b-41c2-8b30-a4e5d7d3afac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 14:33:38.957250 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:38.957211 5060 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 14:33:38.957919 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:38.957268 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" podUID="ae832eba-b22b-41c2-8b30-a4e5d7d3afac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 14:33:39.441778 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.441745 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" Nov 03 14:33:39.455264 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:33:39.455229 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:33:39.552438 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.552396 5060 generic.go:356] "Generic (PLEG): container finished" podID="07fa990f-370e-4d84-b7e6-340f4caa6348" containerID="73f3789700b9b3481bf161f6f8c3a04b171b4965563257f488c66bcdbe9f5452" exitCode=0 Nov 03 14:33:39.552624 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.552458 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" Nov 03 14:33:39.552624 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.552484 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" event={"ID":"07fa990f-370e-4d84-b7e6-340f4caa6348","Type":"ContainerDied","Data":"73f3789700b9b3481bf161f6f8c3a04b171b4965563257f488c66bcdbe9f5452"} Nov 03 14:33:39.552624 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.552551 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj" event={"ID":"07fa990f-370e-4d84-b7e6-340f4caa6348","Type":"ContainerDied","Data":"81b596ceb1af600a150dafabe893671fdc0268e6479df3c014d90a2e3139aa86"} Nov 03 14:33:39.552624 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.552587 5060 scope.go:122] "RemoveContainer" containerID="73f3789700b9b3481bf161f6f8c3a04b171b4965563257f488c66bcdbe9f5452" Nov 03 14:33:39.562953 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.562925 5060 scope.go:122] "RemoveContainer" containerID="f16df8daebf0a320416c241bacc909170a392e9e7159285076f80439fc860ac9" Nov 03 14:33:39.572713 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.572692 5060 scope.go:122] "RemoveContainer" containerID="73f3789700b9b3481bf161f6f8c3a04b171b4965563257f488c66bcdbe9f5452" Nov 03 14:33:39.572971 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.572947 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07fa990f-370e-4d84-b7e6-340f4caa6348-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07fa990f-370e-4d84-b7e6-340f4caa6348-workload-socket\") pod \"07fa990f-370e-4d84-b7e6-340f4caa6348\" (UID: \"07fa990f-370e-4d84-b7e6-340f4caa6348\") " Nov 03 14:33:39.573091 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:33:39.573007 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73f3789700b9b3481bf161f6f8c3a04b171b4965563257f488c66bcdbe9f5452\": container with ID starting with 73f3789700b9b3481bf161f6f8c3a04b171b4965563257f488c66bcdbe9f5452 not found: ID does not exist" containerID="73f3789700b9b3481bf161f6f8c3a04b171b4965563257f488c66bcdbe9f5452" Nov 03 14:33:39.573091 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.573045 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07fa990f-370e-4d84-b7e6-340f4caa6348-kube-api-access-tblzp\" (UniqueName: \"kubernetes.io/projected/07fa990f-370e-4d84-b7e6-340f4caa6348-kube-api-access-tblzp\") pod \"07fa990f-370e-4d84-b7e6-340f4caa6348\" (UID: \"07fa990f-370e-4d84-b7e6-340f4caa6348\") " Nov 03 14:33:39.573181 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.573138 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07fa990f-370e-4d84-b7e6-340f4caa6348-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07fa990f-370e-4d84-b7e6-340f4caa6348-istio-data\") pod \"07fa990f-370e-4d84-b7e6-340f4caa6348\" (UID: \"07fa990f-370e-4d84-b7e6-340f4caa6348\") " Nov 03 14:33:39.573181 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.573043 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f3789700b9b3481bf161f6f8c3a04b171b4965563257f488c66bcdbe9f5452"} err="failed to get container status \"73f3789700b9b3481bf161f6f8c3a04b171b4965563257f488c66bcdbe9f5452\": rpc error: code = NotFound desc = could not find container \"73f3789700b9b3481bf161f6f8c3a04b171b4965563257f488c66bcdbe9f5452\": container with ID starting with 73f3789700b9b3481bf161f6f8c3a04b171b4965563257f488c66bcdbe9f5452 not found: ID does not exist" Nov 03 14:33:39.573181 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.573172 5060 scope.go:122] "RemoveContainer" containerID="f16df8daebf0a320416c241bacc909170a392e9e7159285076f80439fc860ac9" Nov 03 14:33:39.573181 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.573175 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07fa990f-370e-4d84-b7e6-340f4caa6348-istio-token\" (UniqueName: \"kubernetes.io/projected/07fa990f-370e-4d84-b7e6-340f4caa6348-istio-token\") pod \"07fa990f-370e-4d84-b7e6-340f4caa6348\" (UID: \"07fa990f-370e-4d84-b7e6-340f4caa6348\") " Nov 03 14:33:39.573398 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.573212 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07fa990f-370e-4d84-b7e6-340f4caa6348-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07fa990f-370e-4d84-b7e6-340f4caa6348-istio-podinfo\") pod \"07fa990f-370e-4d84-b7e6-340f4caa6348\" (UID: \"07fa990f-370e-4d84-b7e6-340f4caa6348\") " Nov 03 14:33:39.573398 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.573285 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07fa990f-370e-4d84-b7e6-340f4caa6348-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07fa990f-370e-4d84-b7e6-340f4caa6348-istio-envoy\") pod \"07fa990f-370e-4d84-b7e6-340f4caa6348\" (UID: \"07fa990f-370e-4d84-b7e6-340f4caa6348\") " Nov 03 14:33:39.573398 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.573323 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07fa990f-370e-4d84-b7e6-340f4caa6348-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07fa990f-370e-4d84-b7e6-340f4caa6348-credential-socket\") pod \"07fa990f-370e-4d84-b7e6-340f4caa6348\" (UID: \"07fa990f-370e-4d84-b7e6-340f4caa6348\") " Nov 03 14:33:39.573398 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.573350 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07fa990f-370e-4d84-b7e6-340f4caa6348-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07fa990f-370e-4d84-b7e6-340f4caa6348-workload-certs\") pod \"07fa990f-370e-4d84-b7e6-340f4caa6348\" (UID: \"07fa990f-370e-4d84-b7e6-340f4caa6348\") " Nov 03 14:33:39.573607 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.573426 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07fa990f-370e-4d84-b7e6-340f4caa6348-istio-data" pod "07fa990f-370e-4d84-b7e6-340f4caa6348" (UID: "07fa990f-370e-4d84-b7e6-340f4caa6348"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:33:39.573607 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.573476 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07fa990f-370e-4d84-b7e6-340f4caa6348-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07fa990f-370e-4d84-b7e6-340f4caa6348-istiod-ca-cert\") pod \"07fa990f-370e-4d84-b7e6-340f4caa6348\" (UID: \"07fa990f-370e-4d84-b7e6-340f4caa6348\") " Nov 03 14:33:39.573607 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:33:39.573492 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f16df8daebf0a320416c241bacc909170a392e9e7159285076f80439fc860ac9\": container with ID starting with f16df8daebf0a320416c241bacc909170a392e9e7159285076f80439fc860ac9 not found: ID does not exist" containerID="f16df8daebf0a320416c241bacc909170a392e9e7159285076f80439fc860ac9" Nov 03 14:33:39.573607 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.573534 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07fa990f-370e-4d84-b7e6-340f4caa6348-workload-socket" pod "07fa990f-370e-4d84-b7e6-340f4caa6348" (UID: "07fa990f-370e-4d84-b7e6-340f4caa6348"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:33:39.573607 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.573531 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f16df8daebf0a320416c241bacc909170a392e9e7159285076f80439fc860ac9"} err="failed to get container status \"f16df8daebf0a320416c241bacc909170a392e9e7159285076f80439fc860ac9\": rpc error: code = NotFound desc = could not find container \"f16df8daebf0a320416c241bacc909170a392e9e7159285076f80439fc860ac9\": container with ID starting with f16df8daebf0a320416c241bacc909170a392e9e7159285076f80439fc860ac9 not found: ID does not exist" Nov 03 14:33:39.573914 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.573755 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07fa990f-370e-4d84-b7e6-340f4caa6348-credential-socket" pod "07fa990f-370e-4d84-b7e6-340f4caa6348" (UID: "07fa990f-370e-4d84-b7e6-340f4caa6348"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:33:39.573914 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.573786 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07fa990f-370e-4d84-b7e6-340f4caa6348-workload-certs" pod "07fa990f-370e-4d84-b7e6-340f4caa6348" (UID: "07fa990f-370e-4d84-b7e6-340f4caa6348"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:33:39.574029 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.573966 5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07fa990f-370e-4d84-b7e6-340f4caa6348-workload-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:33:39.574029 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.573980 5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07fa990f-370e-4d84-b7e6-340f4caa6348-istio-data\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:33:39.574029 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.573991 5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07fa990f-370e-4d84-b7e6-340f4caa6348-credential-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:33:39.574029 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.574000 5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07fa990f-370e-4d84-b7e6-340f4caa6348-workload-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:33:39.574029 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.574001 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07fa990f-370e-4d84-b7e6-340f4caa6348-istiod-ca-cert" pod "07fa990f-370e-4d84-b7e6-340f4caa6348" (UID: "07fa990f-370e-4d84-b7e6-340f4caa6348"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:33:39.575724 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.575698 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07fa990f-370e-4d84-b7e6-340f4caa6348-istio-envoy" pod "07fa990f-370e-4d84-b7e6-340f4caa6348" (UID: "07fa990f-370e-4d84-b7e6-340f4caa6348"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:33:39.575866 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.575776 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07fa990f-370e-4d84-b7e6-340f4caa6348-istio-token" pod "07fa990f-370e-4d84-b7e6-340f4caa6348" (UID: "07fa990f-370e-4d84-b7e6-340f4caa6348"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:33:39.575944 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.575889 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07fa990f-370e-4d84-b7e6-340f4caa6348-kube-api-access-tblzp" pod "07fa990f-370e-4d84-b7e6-340f4caa6348" (UID: "07fa990f-370e-4d84-b7e6-340f4caa6348"). InnerVolumeSpecName "kube-api-access-tblzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:33:39.576122 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.576096 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07fa990f-370e-4d84-b7e6-340f4caa6348-istio-podinfo" pod "07fa990f-370e-4d84-b7e6-340f4caa6348" (UID: "07fa990f-370e-4d84-b7e6-340f4caa6348"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:33:39.675195 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.675158 5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07fa990f-370e-4d84-b7e6-340f4caa6348-istio-token\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:33:39.675195 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.675189 5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07fa990f-370e-4d84-b7e6-340f4caa6348-istio-podinfo\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:33:39.675195 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.675199 5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07fa990f-370e-4d84-b7e6-340f4caa6348-istio-envoy\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:33:39.675440 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.675208 5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07fa990f-370e-4d84-b7e6-340f4caa6348-istiod-ca-cert\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:33:39.675440 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.675217 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tblzp\" (UniqueName: \"kubernetes.io/projected/07fa990f-370e-4d84-b7e6-340f4caa6348-kube-api-access-tblzp\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:33:39.927968 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.927935 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj"] Nov 03 14:33:39.940605 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.940576 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-gj8jj"] Nov 03 14:33:39.961669 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.961647 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" Nov 03 14:33:39.962051 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.961957 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" Nov 03 14:33:39.962766 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:39.962749 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" Nov 03 14:33:40.043902 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:40.043872 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0db7e167-6hrgl"] Nov 03 14:33:40.047762 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:40.047738 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0db7e167-6hrgl"] Nov 03 14:33:40.251731 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:40.251694 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07fa990f-370e-4d84-b7e6-340f4caa6348" path="/var/lib/kubelet/pods/07fa990f-370e-4d84-b7e6-340f4caa6348/volumes" Nov 03 14:33:40.252360 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:40.252340 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f5389c9-92e4-41ec-bd3f-c4fc4723c51c" path="/var/lib/kubelet/pods/5f5389c9-92e4-41ec-bd3f-c4fc4723c51c/volumes" Nov 03 14:33:40.560288 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:40.560251 5060 generic.go:356] "Generic (PLEG): container finished" podID="599e862d-75d4-40f6-9750-806110c5d308" containerID="eaebb2c149f405314e678a3dae00f4591fcef556ffa1d750490a75466f5f498e" exitCode=0 Nov 03 14:33:40.560480 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:40.560323 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" event={"ID":"599e862d-75d4-40f6-9750-806110c5d308","Type":"ContainerDied","Data":"eaebb2c149f405314e678a3dae00f4591fcef556ffa1d750490a75466f5f498e"} Nov 03 14:33:40.560875 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:40.560826 5060 scope.go:122] "RemoveContainer" containerID="eaebb2c149f405314e678a3dae00f4591fcef556ffa1d750490a75466f5f498e" Nov 03 14:33:40.562282 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:40.562258 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:33:40.562346 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:40.562324 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:33:40.562391 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:40.562354 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:33:41.570231 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:41.570198 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" event={"ID":"599e862d-75d4-40f6-9750-806110c5d308","Type":"ContainerStarted","Data":"fc179af2f0d36f082928a4df1c6ebfedb1c35d5e6a9eb3db8d4cd17c80e186e7"} Nov 03 14:33:41.883349 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:41.883263 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" Nov 03 14:33:41.884531 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:41.884505 5060 patch_prober.go:29] interesting pod/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 14:33:41.884644 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:41.884552 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" podUID="599e862d-75d4-40f6-9750-806110c5d308" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 14:33:42.018984 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:42.018944 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" Nov 03 14:33:42.883580 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:42.883544 5060 patch_prober.go:29] interesting pod/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 14:33:42.884015 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:42.883599 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" podUID="599e862d-75d4-40f6-9750-806110c5d308" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 14:33:43.883497 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:43.883455 5060 patch_prober.go:29] interesting pod/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 14:33:43.883709 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:43.883514 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" podUID="599e862d-75d4-40f6-9750-806110c5d308" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 14:33:44.045979 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:44.045937 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0e0e478-hmwl5"] Nov 03 14:33:44.058381 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:44.058349 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0e0e478-hmwl5"] Nov 03 14:33:44.251273 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:44.251235 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dc68081-80ec-4fc5-81c8-ec0b96a4647a" path="/var/lib/kubelet/pods/9dc68081-80ec-4fc5-81c8-ec0b96a4647a/volumes" Nov 03 14:33:44.883668 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:44.883629 5060 patch_prober.go:29] interesting pod/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 14:33:44.883883 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:44.883683 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" podUID="599e862d-75d4-40f6-9750-806110c5d308" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 14:33:45.883351 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:45.883317 5060 patch_prober.go:29] interesting pod/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 14:33:45.883544 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:45.883438 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" podUID="599e862d-75d4-40f6-9750-806110c5d308" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 14:33:46.901368 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:46.901339 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" Nov 03 14:33:46.902566 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:46.902544 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" Nov 03 14:33:55.052775 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:55.052731 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-957aebcd-dpbfp"] Nov 03 14:33:55.058943 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:55.058916 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-957aebcd-dpbfp"] Nov 03 14:33:56.252591 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:56.252557 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0877646-7ea1-4320-b6de-888cfd2fdce7" path="/var/lib/kubelet/pods/e0877646-7ea1-4320-b6de-888cfd2fdce7/volumes" Nov 03 14:33:56.949660 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:56.949625 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-95af53a1-bd758ddff-zngcj"] Nov 03 14:33:56.949930 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:56.949880 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-95af53a1-bd758ddff-zngcj" podUID="957440c5-6bdc-42eb-ab1e-9e78e026ca43" containerName="maas-api" containerID="cri-o://bd53ee25ffc52cdb9b52d7f04647ba33e9dae64e4dfdd27c24f023c4eaf1da5e" gracePeriod=30 Nov 03 14:33:57.302903 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:57.302817 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-95af53a1-bd758ddff-zngcj" Nov 03 14:33:57.398602 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:57.398568 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/957440c5-6bdc-42eb-ab1e-9e78e026ca43-kube-api-access-lzwsw\" (UniqueName: \"kubernetes.io/projected/957440c5-6bdc-42eb-ab1e-9e78e026ca43-kube-api-access-lzwsw\") pod \"957440c5-6bdc-42eb-ab1e-9e78e026ca43\" (UID: \"957440c5-6bdc-42eb-ab1e-9e78e026ca43\") " Nov 03 14:33:57.398796 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:57.398612 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/957440c5-6bdc-42eb-ab1e-9e78e026ca43-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/957440c5-6bdc-42eb-ab1e-9e78e026ca43-openshift-service-ca\") pod \"957440c5-6bdc-42eb-ab1e-9e78e026ca43\" (UID: \"957440c5-6bdc-42eb-ab1e-9e78e026ca43\") " Nov 03 14:33:57.398796 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:57.398661 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/957440c5-6bdc-42eb-ab1e-9e78e026ca43-metrics-certs\" (UniqueName: \"kubernetes.io/secret/957440c5-6bdc-42eb-ab1e-9e78e026ca43-metrics-certs\") pod \"957440c5-6bdc-42eb-ab1e-9e78e026ca43\" (UID: \"957440c5-6bdc-42eb-ab1e-9e78e026ca43\") " Nov 03 14:33:57.398796 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:57.398711 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/957440c5-6bdc-42eb-ab1e-9e78e026ca43-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/957440c5-6bdc-42eb-ab1e-9e78e026ca43-maas-api-tls\") pod \"957440c5-6bdc-42eb-ab1e-9e78e026ca43\" (UID: \"957440c5-6bdc-42eb-ab1e-9e78e026ca43\") " Nov 03 14:33:57.400344 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:57.400308 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/957440c5-6bdc-42eb-ab1e-9e78e026ca43-openshift-service-ca" pod "957440c5-6bdc-42eb-ab1e-9e78e026ca43" (UID: "957440c5-6bdc-42eb-ab1e-9e78e026ca43"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:33:57.401014 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:57.400991 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/957440c5-6bdc-42eb-ab1e-9e78e026ca43-kube-api-access-lzwsw" pod "957440c5-6bdc-42eb-ab1e-9e78e026ca43" (UID: "957440c5-6bdc-42eb-ab1e-9e78e026ca43"). InnerVolumeSpecName "kube-api-access-lzwsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:33:57.401192 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:57.401169 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/957440c5-6bdc-42eb-ab1e-9e78e026ca43-metrics-certs" pod "957440c5-6bdc-42eb-ab1e-9e78e026ca43" (UID: "957440c5-6bdc-42eb-ab1e-9e78e026ca43"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:33:57.402521 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:57.402501 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/957440c5-6bdc-42eb-ab1e-9e78e026ca43-maas-api-tls" pod "957440c5-6bdc-42eb-ab1e-9e78e026ca43" (UID: "957440c5-6bdc-42eb-ab1e-9e78e026ca43"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:33:57.520932 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:57.518033 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzwsw\" (UniqueName: \"kubernetes.io/projected/957440c5-6bdc-42eb-ab1e-9e78e026ca43-kube-api-access-lzwsw\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:33:57.520932 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:57.518081 5060 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/957440c5-6bdc-42eb-ab1e-9e78e026ca43-openshift-service-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:33:57.520932 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:57.518100 5060 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/957440c5-6bdc-42eb-ab1e-9e78e026ca43-metrics-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:33:57.520932 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:57.518118 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/957440c5-6bdc-42eb-ab1e-9e78e026ca43-maas-api-tls\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:33:57.719663 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:57.719627 5060 generic.go:356] "Generic (PLEG): container finished" podID="957440c5-6bdc-42eb-ab1e-9e78e026ca43" containerID="bd53ee25ffc52cdb9b52d7f04647ba33e9dae64e4dfdd27c24f023c4eaf1da5e" exitCode=0 Nov 03 14:33:57.719880 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:57.719678 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-95af53a1-bd758ddff-zngcj" Nov 03 14:33:57.719880 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:57.719710 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-95af53a1-bd758ddff-zngcj" event={"ID":"957440c5-6bdc-42eb-ab1e-9e78e026ca43","Type":"ContainerDied","Data":"bd53ee25ffc52cdb9b52d7f04647ba33e9dae64e4dfdd27c24f023c4eaf1da5e"} Nov 03 14:33:57.719880 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:57.719764 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-95af53a1-bd758ddff-zngcj" event={"ID":"957440c5-6bdc-42eb-ab1e-9e78e026ca43","Type":"ContainerDied","Data":"026ae8cc816ea9be8070f90ed54d0a1f353a3e897fe9413f3973594987869b55"} Nov 03 14:33:57.719880 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:57.719786 5060 scope.go:122] "RemoveContainer" containerID="bd53ee25ffc52cdb9b52d7f04647ba33e9dae64e4dfdd27c24f023c4eaf1da5e" Nov 03 14:33:57.730792 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:57.730774 5060 scope.go:122] "RemoveContainer" containerID="bd53ee25ffc52cdb9b52d7f04647ba33e9dae64e4dfdd27c24f023c4eaf1da5e" Nov 03 14:33:57.731079 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:33:57.731061 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd53ee25ffc52cdb9b52d7f04647ba33e9dae64e4dfdd27c24f023c4eaf1da5e\": container with ID starting with bd53ee25ffc52cdb9b52d7f04647ba33e9dae64e4dfdd27c24f023c4eaf1da5e not found: ID does not exist" containerID="bd53ee25ffc52cdb9b52d7f04647ba33e9dae64e4dfdd27c24f023c4eaf1da5e" Nov 03 14:33:57.731123 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:57.731088 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd53ee25ffc52cdb9b52d7f04647ba33e9dae64e4dfdd27c24f023c4eaf1da5e"} err="failed to get container status \"bd53ee25ffc52cdb9b52d7f04647ba33e9dae64e4dfdd27c24f023c4eaf1da5e\": rpc error: code = NotFound desc = could not find container \"bd53ee25ffc52cdb9b52d7f04647ba33e9dae64e4dfdd27c24f023c4eaf1da5e\": container with ID starting with bd53ee25ffc52cdb9b52d7f04647ba33e9dae64e4dfdd27c24f023c4eaf1da5e not found: ID does not exist" Nov 03 14:33:57.777622 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:57.777552 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-95af53a1-bd758ddff-zngcj"] Nov 03 14:33:57.790275 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:57.790247 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-95af53a1-bd758ddff-zngcj"] Nov 03 14:33:58.112337 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:58.112247 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j"] Nov 03 14:33:58.112572 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:58.112545 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" podUID="599e862d-75d4-40f6-9750-806110c5d308" containerName="istio-proxy" containerID="cri-o://fc179af2f0d36f082928a4df1c6ebfedb1c35d5e6a9eb3db8d4cd17c80e186e7" gracePeriod=30 Nov 03 14:33:58.256454 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:58.256409 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="957440c5-6bdc-42eb-ab1e-9e78e026ca43" path="/var/lib/kubelet/pods/957440c5-6bdc-42eb-ab1e-9e78e026ca43/volumes" Nov 03 14:33:59.332538 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:59.332503 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc"] Nov 03 14:33:59.333859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:59.333812 5060 cpu_manager.go:414] "containerMap: removing container" podUID="07fa990f-370e-4d84-b7e6-340f4caa6348" containerName="istio-proxy" Nov 03 14:33:59.334017 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:59.334002 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07fa990f-370e-4d84-b7e6-340f4caa6348" containerName="istio-proxy" Nov 03 14:33:59.334146 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:59.334135 5060 cpu_manager.go:414] "containerMap: removing container" podUID="957440c5-6bdc-42eb-ab1e-9e78e026ca43" containerName="maas-api" Nov 03 14:33:59.334222 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:59.334212 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="957440c5-6bdc-42eb-ab1e-9e78e026ca43" containerName="maas-api" Nov 03 14:33:59.334476 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:59.334463 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="07fa990f-370e-4d84-b7e6-340f4caa6348" containerName="istio-proxy" Nov 03 14:33:59.334562 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:59.334551 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="957440c5-6bdc-42eb-ab1e-9e78e026ca43" containerName="maas-api" Nov 03 14:33:59.339899 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:59.339878 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" Nov 03 14:33:59.360247 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:59.360220 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-cb587c-openshift-default-dockercfg-gq2bk\"" Nov 03 14:33:59.393926 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:59.393878 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc"] Nov 03 14:33:59.442290 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:59.442258 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dc088f1-c679-4845-ba33-682ed25b6d22-istio-token\") pod \"e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc\" (UID: \"8dc088f1-c679-4845-ba33-682ed25b6d22\") " pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" Nov 03 14:33:59.442290 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:59.442301 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gdsq\" (UniqueName: \"kubernetes.io/projected/8dc088f1-c679-4845-ba33-682ed25b6d22-kube-api-access-8gdsq\") pod \"e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc\" (UID: \"8dc088f1-c679-4845-ba33-682ed25b6d22\") " pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" Nov 03 14:33:59.442495 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:59.442332 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dc088f1-c679-4845-ba33-682ed25b6d22-istio-data\") pod \"e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc\" (UID: \"8dc088f1-c679-4845-ba33-682ed25b6d22\") " pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" Nov 03 14:33:59.442495 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:59.442425 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dc088f1-c679-4845-ba33-682ed25b6d22-istio-envoy\") pod \"e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc\" (UID: \"8dc088f1-c679-4845-ba33-682ed25b6d22\") " pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" Nov 03 14:33:59.442495 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:59.442464 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dc088f1-c679-4845-ba33-682ed25b6d22-istio-podinfo\") pod \"e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc\" (UID: \"8dc088f1-c679-4845-ba33-682ed25b6d22\") " pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" Nov 03 14:33:59.442591 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:59.442497 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dc088f1-c679-4845-ba33-682ed25b6d22-workload-socket\") pod \"e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc\" (UID: \"8dc088f1-c679-4845-ba33-682ed25b6d22\") " pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" Nov 03 14:33:59.442654 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:59.442640 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dc088f1-c679-4845-ba33-682ed25b6d22-credential-socket\") pod \"e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc\" (UID: \"8dc088f1-c679-4845-ba33-682ed25b6d22\") " pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" Nov 03 14:33:59.442689 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:59.442671 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dc088f1-c679-4845-ba33-682ed25b6d22-workload-certs\") pod \"e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc\" (UID: \"8dc088f1-c679-4845-ba33-682ed25b6d22\") " pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" Nov 03 14:33:59.442776 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:59.442758 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dc088f1-c679-4845-ba33-682ed25b6d22-istiod-ca-cert\") pod \"e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc\" (UID: \"8dc088f1-c679-4845-ba33-682ed25b6d22\") " pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" Nov 03 14:33:59.597269 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:59.560599 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dc088f1-c679-4845-ba33-682ed25b6d22-credential-socket\") pod \"e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc\" (UID: \"8dc088f1-c679-4845-ba33-682ed25b6d22\") " pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" Nov 03 14:33:59.597269 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:59.560666 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dc088f1-c679-4845-ba33-682ed25b6d22-workload-certs\") pod \"e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc\" (UID: \"8dc088f1-c679-4845-ba33-682ed25b6d22\") " pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" Nov 03 14:33:59.597269 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:59.560746 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dc088f1-c679-4845-ba33-682ed25b6d22-istiod-ca-cert\") pod \"e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc\" (UID: \"8dc088f1-c679-4845-ba33-682ed25b6d22\") " pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" Nov 03 14:33:59.597269 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:59.560780 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dc088f1-c679-4845-ba33-682ed25b6d22-istio-token\") pod \"e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc\" (UID: \"8dc088f1-c679-4845-ba33-682ed25b6d22\") " pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" Nov 03 14:33:59.597269 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:59.560816 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gdsq\" (UniqueName: \"kubernetes.io/projected/8dc088f1-c679-4845-ba33-682ed25b6d22-kube-api-access-8gdsq\") pod \"e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc\" (UID: \"8dc088f1-c679-4845-ba33-682ed25b6d22\") " pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" Nov 03 14:33:59.597269 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:59.560867 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dc088f1-c679-4845-ba33-682ed25b6d22-istio-data\") pod \"e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc\" (UID: \"8dc088f1-c679-4845-ba33-682ed25b6d22\") " pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" Nov 03 14:33:59.597269 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:59.560909 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dc088f1-c679-4845-ba33-682ed25b6d22-istio-envoy\") pod \"e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc\" (UID: \"8dc088f1-c679-4845-ba33-682ed25b6d22\") " pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" Nov 03 14:33:59.597269 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:59.560970 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dc088f1-c679-4845-ba33-682ed25b6d22-istio-podinfo\") pod \"e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc\" (UID: \"8dc088f1-c679-4845-ba33-682ed25b6d22\") " pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" Nov 03 14:33:59.597269 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:59.561011 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dc088f1-c679-4845-ba33-682ed25b6d22-workload-socket\") pod \"e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc\" (UID: \"8dc088f1-c679-4845-ba33-682ed25b6d22\") " pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" Nov 03 14:33:59.597269 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:59.561600 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dc088f1-c679-4845-ba33-682ed25b6d22-workload-socket\") pod \"e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc\" (UID: \"8dc088f1-c679-4845-ba33-682ed25b6d22\") " pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" Nov 03 14:33:59.597269 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:59.561875 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dc088f1-c679-4845-ba33-682ed25b6d22-credential-socket\") pod \"e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc\" (UID: \"8dc088f1-c679-4845-ba33-682ed25b6d22\") " pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" Nov 03 14:33:59.597269 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:59.562127 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dc088f1-c679-4845-ba33-682ed25b6d22-workload-certs\") pod \"e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc\" (UID: \"8dc088f1-c679-4845-ba33-682ed25b6d22\") " pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" Nov 03 14:33:59.597269 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:59.562754 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dc088f1-c679-4845-ba33-682ed25b6d22-istiod-ca-cert\") pod \"e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc\" (UID: \"8dc088f1-c679-4845-ba33-682ed25b6d22\") " pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" Nov 03 14:33:59.597269 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:59.563513 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dc088f1-c679-4845-ba33-682ed25b6d22-istio-data\") pod \"e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc\" (UID: \"8dc088f1-c679-4845-ba33-682ed25b6d22\") " pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" Nov 03 14:33:59.598482 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:59.598383 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dc088f1-c679-4845-ba33-682ed25b6d22-istio-envoy\") pod \"e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc\" (UID: \"8dc088f1-c679-4845-ba33-682ed25b6d22\") " pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" Nov 03 14:33:59.679106 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:59.679034 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gdsq\" (UniqueName: \"kubernetes.io/projected/8dc088f1-c679-4845-ba33-682ed25b6d22-kube-api-access-8gdsq\") pod \"e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc\" (UID: \"8dc088f1-c679-4845-ba33-682ed25b6d22\") " pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" Nov 03 14:33:59.688635 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:59.686698 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dc088f1-c679-4845-ba33-682ed25b6d22-istio-token\") pod \"e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc\" (UID: \"8dc088f1-c679-4845-ba33-682ed25b6d22\") " pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" Nov 03 14:33:59.691855 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:59.689376 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dc088f1-c679-4845-ba33-682ed25b6d22-istio-podinfo\") pod \"e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc\" (UID: \"8dc088f1-c679-4845-ba33-682ed25b6d22\") " pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" Nov 03 14:33:59.961713 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:33:59.961624 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" Nov 03 14:34:00.300546 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:00.292723 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc"] Nov 03 14:34:00.303044 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:00.303001 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:34:00.303140 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:00.303085 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:34:00.303140 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:00.303123 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:34:00.743693 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:00.743656 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" event={"ID":"8dc088f1-c679-4845-ba33-682ed25b6d22","Type":"ContainerStarted","Data":"86a3d9c1575232449cbfdcb3b93c0988085e6bdbc41e660126bdb4cfd6ef9af0"} Nov 03 14:34:00.744077 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:00.743709 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" event={"ID":"8dc088f1-c679-4845-ba33-682ed25b6d22","Type":"ContainerStarted","Data":"359cb1ec5e61383b6987fb927b19b997f58f97777673212e2618626924081e05"} Nov 03 14:34:00.813591 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:00.813529 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" podStartSLOduration=1.813511649 podStartE2EDuration="1.813511649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:34:00.810980448 +0000 UTC m=+1374.699770029" watchObservedRunningTime="2026-11-03 14:34:00.813511649 +0000 UTC m=+1374.702301230" Nov 03 14:34:00.961980 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:00.961934 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" Nov 03 14:34:01.050582 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:01.050488 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-238b8528-spcqq"] Nov 03 14:34:01.057385 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:01.057353 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-238b8528-spcqq"] Nov 03 14:34:01.904938 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:01.904903 5060 patch_prober.go:29] interesting pod/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 14:34:01.905342 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:01.904955 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" podUID="599e862d-75d4-40f6-9750-806110c5d308" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 14:34:01.939940 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:01.939893 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-6653eb-59f9f85797-68tkt"] Nov 03 14:34:01.940151 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:01.940125 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6653eb-59f9f85797-68tkt" podUID="74a86163-4d09-49e1-945d-521f611bda7e" containerName="maas-api" containerID="cri-o://e8f36f62e2407ecb1636fc01fe6cba2f786f75cd70dc3c225a7b923fc4199eb3" gracePeriod=30 Nov 03 14:34:01.967665 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:01.967637 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" Nov 03 14:34:02.253466 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:02.253415 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcfd1798-5f57-4e1c-b57e-8f4101883025" path="/var/lib/kubelet/pods/fcfd1798-5f57-4e1c-b57e-8f4101883025/volumes" Nov 03 14:34:02.274683 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:02.274657 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6653eb-59f9f85797-68tkt" Nov 03 14:34:02.312980 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:02.312941 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74a86163-4d09-49e1-945d-521f611bda7e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74a86163-4d09-49e1-945d-521f611bda7e-openshift-service-ca\") pod \"74a86163-4d09-49e1-945d-521f611bda7e\" (UID: \"74a86163-4d09-49e1-945d-521f611bda7e\") " Nov 03 14:34:02.313129 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:02.312993 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74a86163-4d09-49e1-945d-521f611bda7e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/74a86163-4d09-49e1-945d-521f611bda7e-metrics-certs\") pod \"74a86163-4d09-49e1-945d-521f611bda7e\" (UID: \"74a86163-4d09-49e1-945d-521f611bda7e\") " Nov 03 14:34:02.313129 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:02.313044 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74a86163-4d09-49e1-945d-521f611bda7e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74a86163-4d09-49e1-945d-521f611bda7e-maas-api-tls\") pod \"74a86163-4d09-49e1-945d-521f611bda7e\" (UID: \"74a86163-4d09-49e1-945d-521f611bda7e\") " Nov 03 14:34:02.313225 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:02.313133 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74a86163-4d09-49e1-945d-521f611bda7e-kube-api-access-pcwmc\" (UniqueName: \"kubernetes.io/projected/74a86163-4d09-49e1-945d-521f611bda7e-kube-api-access-pcwmc\") pod \"74a86163-4d09-49e1-945d-521f611bda7e\" (UID: \"74a86163-4d09-49e1-945d-521f611bda7e\") " Nov 03 14:34:02.313374 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:02.313344 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74a86163-4d09-49e1-945d-521f611bda7e-openshift-service-ca" pod "74a86163-4d09-49e1-945d-521f611bda7e" (UID: "74a86163-4d09-49e1-945d-521f611bda7e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:34:02.313719 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:02.313698 5060 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74a86163-4d09-49e1-945d-521f611bda7e-openshift-service-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:34:02.315286 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:02.315261 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74a86163-4d09-49e1-945d-521f611bda7e-metrics-certs" pod "74a86163-4d09-49e1-945d-521f611bda7e" (UID: "74a86163-4d09-49e1-945d-521f611bda7e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:34:02.315425 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:02.315296 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a86163-4d09-49e1-945d-521f611bda7e-kube-api-access-pcwmc" pod "74a86163-4d09-49e1-945d-521f611bda7e" (UID: "74a86163-4d09-49e1-945d-521f611bda7e"). InnerVolumeSpecName "kube-api-access-pcwmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:34:02.315495 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:02.315424 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74a86163-4d09-49e1-945d-521f611bda7e-maas-api-tls" pod "74a86163-4d09-49e1-945d-521f611bda7e" (UID: "74a86163-4d09-49e1-945d-521f611bda7e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:34:02.415375 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:02.415341 5060 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74a86163-4d09-49e1-945d-521f611bda7e-metrics-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:34:02.415375 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:02.415373 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74a86163-4d09-49e1-945d-521f611bda7e-maas-api-tls\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:34:02.415375 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:02.415383 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcwmc\" (UniqueName: \"kubernetes.io/projected/74a86163-4d09-49e1-945d-521f611bda7e-kube-api-access-pcwmc\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:34:02.758160 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:02.758124 5060 generic.go:356] "Generic (PLEG): container finished" podID="74a86163-4d09-49e1-945d-521f611bda7e" containerID="e8f36f62e2407ecb1636fc01fe6cba2f786f75cd70dc3c225a7b923fc4199eb3" exitCode=0 Nov 03 14:34:02.758363 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:02.758169 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6653eb-59f9f85797-68tkt" Nov 03 14:34:02.758363 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:02.758212 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6653eb-59f9f85797-68tkt" event={"ID":"74a86163-4d09-49e1-945d-521f611bda7e","Type":"ContainerDied","Data":"e8f36f62e2407ecb1636fc01fe6cba2f786f75cd70dc3c225a7b923fc4199eb3"} Nov 03 14:34:02.758363 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:02.758271 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6653eb-59f9f85797-68tkt" event={"ID":"74a86163-4d09-49e1-945d-521f611bda7e","Type":"ContainerDied","Data":"b573f04cd163cf8a3b9699024391a05e55168135b5e1cbda597eba1798baaa47"} Nov 03 14:34:02.758363 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:02.758302 5060 scope.go:122] "RemoveContainer" containerID="e8f36f62e2407ecb1636fc01fe6cba2f786f75cd70dc3c225a7b923fc4199eb3" Nov 03 14:34:02.758601 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:02.758585 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" Nov 03 14:34:02.759940 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:02.759919 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" Nov 03 14:34:02.769582 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:02.769562 5060 scope.go:122] "RemoveContainer" containerID="e8f36f62e2407ecb1636fc01fe6cba2f786f75cd70dc3c225a7b923fc4199eb3" Nov 03 14:34:02.769880 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:34:02.769854 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8f36f62e2407ecb1636fc01fe6cba2f786f75cd70dc3c225a7b923fc4199eb3\": container with ID starting with e8f36f62e2407ecb1636fc01fe6cba2f786f75cd70dc3c225a7b923fc4199eb3 not found: ID does not exist" containerID="e8f36f62e2407ecb1636fc01fe6cba2f786f75cd70dc3c225a7b923fc4199eb3" Nov 03 14:34:02.769949 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:02.769892 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8f36f62e2407ecb1636fc01fe6cba2f786f75cd70dc3c225a7b923fc4199eb3"} err="failed to get container status \"e8f36f62e2407ecb1636fc01fe6cba2f786f75cd70dc3c225a7b923fc4199eb3\": rpc error: code = NotFound desc = could not find container \"e8f36f62e2407ecb1636fc01fe6cba2f786f75cd70dc3c225a7b923fc4199eb3\": container with ID starting with e8f36f62e2407ecb1636fc01fe6cba2f786f75cd70dc3c225a7b923fc4199eb3 not found: ID does not exist" Nov 03 14:34:02.816666 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:02.816631 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-6653eb-59f9f85797-68tkt"] Nov 03 14:34:02.829297 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:02.829269 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-6653eb-59f9f85797-68tkt"] Nov 03 14:34:03.414746 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:03.414715 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" Nov 03 14:34:03.529724 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:03.529689 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/599e862d-75d4-40f6-9750-806110c5d308-istio-token\" (UniqueName: \"kubernetes.io/projected/599e862d-75d4-40f6-9750-806110c5d308-istio-token\") pod \"599e862d-75d4-40f6-9750-806110c5d308\" (UID: \"599e862d-75d4-40f6-9750-806110c5d308\") " Nov 03 14:34:03.530046 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:03.529826 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/599e862d-75d4-40f6-9750-806110c5d308-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/599e862d-75d4-40f6-9750-806110c5d308-istio-envoy\") pod \"599e862d-75d4-40f6-9750-806110c5d308\" (UID: \"599e862d-75d4-40f6-9750-806110c5d308\") " Nov 03 14:34:03.530046 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:03.529879 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/599e862d-75d4-40f6-9750-806110c5d308-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/599e862d-75d4-40f6-9750-806110c5d308-workload-certs\") pod \"599e862d-75d4-40f6-9750-806110c5d308\" (UID: \"599e862d-75d4-40f6-9750-806110c5d308\") " Nov 03 14:34:03.530046 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:03.529943 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/599e862d-75d4-40f6-9750-806110c5d308-kube-api-access-8n7sl\" (UniqueName: \"kubernetes.io/projected/599e862d-75d4-40f6-9750-806110c5d308-kube-api-access-8n7sl\") pod \"599e862d-75d4-40f6-9750-806110c5d308\" (UID: \"599e862d-75d4-40f6-9750-806110c5d308\") " Nov 03 14:34:03.530046 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:03.530005 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/599e862d-75d4-40f6-9750-806110c5d308-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/599e862d-75d4-40f6-9750-806110c5d308-workload-socket\") pod \"599e862d-75d4-40f6-9750-806110c5d308\" (UID: \"599e862d-75d4-40f6-9750-806110c5d308\") " Nov 03 14:34:03.530046 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:03.530028 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/599e862d-75d4-40f6-9750-806110c5d308-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/599e862d-75d4-40f6-9750-806110c5d308-credential-socket\") pod \"599e862d-75d4-40f6-9750-806110c5d308\" (UID: \"599e862d-75d4-40f6-9750-806110c5d308\") " Nov 03 14:34:03.530321 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:03.530066 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/599e862d-75d4-40f6-9750-806110c5d308-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/599e862d-75d4-40f6-9750-806110c5d308-istio-data\") pod \"599e862d-75d4-40f6-9750-806110c5d308\" (UID: \"599e862d-75d4-40f6-9750-806110c5d308\") " Nov 03 14:34:03.530321 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:03.530143 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/599e862d-75d4-40f6-9750-806110c5d308-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/599e862d-75d4-40f6-9750-806110c5d308-istio-podinfo\") pod \"599e862d-75d4-40f6-9750-806110c5d308\" (UID: \"599e862d-75d4-40f6-9750-806110c5d308\") " Nov 03 14:34:03.530321 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:03.530181 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/599e862d-75d4-40f6-9750-806110c5d308-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/599e862d-75d4-40f6-9750-806110c5d308-istiod-ca-cert\") pod \"599e862d-75d4-40f6-9750-806110c5d308\" (UID: \"599e862d-75d4-40f6-9750-806110c5d308\") " Nov 03 14:34:03.530453 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:03.530376 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/599e862d-75d4-40f6-9750-806110c5d308-istio-data" pod "599e862d-75d4-40f6-9750-806110c5d308" (UID: "599e862d-75d4-40f6-9750-806110c5d308"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:34:03.530689 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:03.530663 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/599e862d-75d4-40f6-9750-806110c5d308-credential-socket" pod "599e862d-75d4-40f6-9750-806110c5d308" (UID: "599e862d-75d4-40f6-9750-806110c5d308"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:34:03.530784 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:03.530680 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/599e862d-75d4-40f6-9750-806110c5d308-workload-socket" pod "599e862d-75d4-40f6-9750-806110c5d308" (UID: "599e862d-75d4-40f6-9750-806110c5d308"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:34:03.530784 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:03.530734 5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/599e862d-75d4-40f6-9750-806110c5d308-istio-data\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:34:03.530990 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:03.530969 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/599e862d-75d4-40f6-9750-806110c5d308-istiod-ca-cert" pod "599e862d-75d4-40f6-9750-806110c5d308" (UID: "599e862d-75d4-40f6-9750-806110c5d308"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:34:03.531055 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:03.530975 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/599e862d-75d4-40f6-9750-806110c5d308-workload-certs" pod "599e862d-75d4-40f6-9750-806110c5d308" (UID: "599e862d-75d4-40f6-9750-806110c5d308"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:34:03.532617 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:03.532554 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/599e862d-75d4-40f6-9750-806110c5d308-istio-podinfo" pod "599e862d-75d4-40f6-9750-806110c5d308" (UID: "599e862d-75d4-40f6-9750-806110c5d308"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:34:03.532731 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:03.532647 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/599e862d-75d4-40f6-9750-806110c5d308-istio-envoy" pod "599e862d-75d4-40f6-9750-806110c5d308" (UID: "599e862d-75d4-40f6-9750-806110c5d308"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:34:03.533023 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:03.533003 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/599e862d-75d4-40f6-9750-806110c5d308-istio-token" pod "599e862d-75d4-40f6-9750-806110c5d308" (UID: "599e862d-75d4-40f6-9750-806110c5d308"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:34:03.533283 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:03.533267 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/599e862d-75d4-40f6-9750-806110c5d308-kube-api-access-8n7sl" pod "599e862d-75d4-40f6-9750-806110c5d308" (UID: "599e862d-75d4-40f6-9750-806110c5d308"). InnerVolumeSpecName "kube-api-access-8n7sl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:34:03.632571 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:03.632541 5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/599e862d-75d4-40f6-9750-806110c5d308-istio-token\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:34:03.632571 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:03.632569 5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/599e862d-75d4-40f6-9750-806110c5d308-istio-envoy\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:34:03.632571 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:03.632578 5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/599e862d-75d4-40f6-9750-806110c5d308-workload-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:34:03.632804 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:03.632586 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n7sl\" (UniqueName: \"kubernetes.io/projected/599e862d-75d4-40f6-9750-806110c5d308-kube-api-access-8n7sl\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:34:03.632804 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:03.632596 5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/599e862d-75d4-40f6-9750-806110c5d308-workload-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:34:03.632804 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:03.632604 5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/599e862d-75d4-40f6-9750-806110c5d308-credential-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:34:03.632804 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:03.632613 5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/599e862d-75d4-40f6-9750-806110c5d308-istio-podinfo\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:34:03.632804 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:03.632621 5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/599e862d-75d4-40f6-9750-806110c5d308-istiod-ca-cert\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:34:03.770261 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:03.770227 5060 generic.go:356] "Generic (PLEG): container finished" podID="599e862d-75d4-40f6-9750-806110c5d308" containerID="fc179af2f0d36f082928a4df1c6ebfedb1c35d5e6a9eb3db8d4cd17c80e186e7" exitCode=0 Nov 03 14:34:03.770446 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:03.770292 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" Nov 03 14:34:03.770446 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:03.770319 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" event={"ID":"599e862d-75d4-40f6-9750-806110c5d308","Type":"ContainerDied","Data":"fc179af2f0d36f082928a4df1c6ebfedb1c35d5e6a9eb3db8d4cd17c80e186e7"} Nov 03 14:34:03.770446 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:03.770380 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j" event={"ID":"599e862d-75d4-40f6-9750-806110c5d308","Type":"ContainerDied","Data":"77ffade70e8d636ee736a10d16798ea286568c98f8d3ce09d594f40e2961f00c"} Nov 03 14:34:03.770446 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:03.770410 5060 scope.go:122] "RemoveContainer" containerID="fc179af2f0d36f082928a4df1c6ebfedb1c35d5e6a9eb3db8d4cd17c80e186e7" Nov 03 14:34:03.781928 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:03.781822 5060 scope.go:122] "RemoveContainer" containerID="eaebb2c149f405314e678a3dae00f4591fcef556ffa1d750490a75466f5f498e" Nov 03 14:34:03.791867 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:03.791821 5060 scope.go:122] "RemoveContainer" containerID="fc179af2f0d36f082928a4df1c6ebfedb1c35d5e6a9eb3db8d4cd17c80e186e7" Nov 03 14:34:03.792183 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:34:03.792161 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc179af2f0d36f082928a4df1c6ebfedb1c35d5e6a9eb3db8d4cd17c80e186e7\": container with ID starting with fc179af2f0d36f082928a4df1c6ebfedb1c35d5e6a9eb3db8d4cd17c80e186e7 not found: ID does not exist" containerID="fc179af2f0d36f082928a4df1c6ebfedb1c35d5e6a9eb3db8d4cd17c80e186e7" Nov 03 14:34:03.792243 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:03.792193 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc179af2f0d36f082928a4df1c6ebfedb1c35d5e6a9eb3db8d4cd17c80e186e7"} err="failed to get container status \"fc179af2f0d36f082928a4df1c6ebfedb1c35d5e6a9eb3db8d4cd17c80e186e7\": rpc error: code = NotFound desc = could not find container \"fc179af2f0d36f082928a4df1c6ebfedb1c35d5e6a9eb3db8d4cd17c80e186e7\": container with ID starting with fc179af2f0d36f082928a4df1c6ebfedb1c35d5e6a9eb3db8d4cd17c80e186e7 not found: ID does not exist" Nov 03 14:34:03.792243 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:03.792211 5060 scope.go:122] "RemoveContainer" containerID="eaebb2c149f405314e678a3dae00f4591fcef556ffa1d750490a75466f5f498e" Nov 03 14:34:03.792513 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:34:03.792490 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaebb2c149f405314e678a3dae00f4591fcef556ffa1d750490a75466f5f498e\": container with ID starting with eaebb2c149f405314e678a3dae00f4591fcef556ffa1d750490a75466f5f498e not found: ID does not exist" containerID="eaebb2c149f405314e678a3dae00f4591fcef556ffa1d750490a75466f5f498e" Nov 03 14:34:03.792571 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:03.792520 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaebb2c149f405314e678a3dae00f4591fcef556ffa1d750490a75466f5f498e"} err="failed to get container status \"eaebb2c149f405314e678a3dae00f4591fcef556ffa1d750490a75466f5f498e\": rpc error: code = NotFound desc = could not find container \"eaebb2c149f405314e678a3dae00f4591fcef556ffa1d750490a75466f5f498e\": container with ID starting with eaebb2c149f405314e678a3dae00f4591fcef556ffa1d750490a75466f5f498e not found: ID does not exist" Nov 03 14:34:03.829603 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:03.829557 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j"] Nov 03 14:34:03.843079 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:03.843049 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-95af53a1-openshift-default-7f85f57d7c-c2x5j"] Nov 03 14:34:04.249546 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:04.249516 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="599e862d-75d4-40f6-9750-806110c5d308" path="/var/lib/kubelet/pods/599e862d-75d4-40f6-9750-806110c5d308/volumes" Nov 03 14:34:04.250020 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:04.250005 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74a86163-4d09-49e1-945d-521f611bda7e" path="/var/lib/kubelet/pods/74a86163-4d09-49e1-945d-521f611bda7e/volumes" Nov 03 14:34:07.357984 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:07.357688 5060 scope.go:122] "RemoveContainer" containerID="74e4d3c483f51d6ece91b9736dc8f415f394130ad3861af4c91686cbc7ee1f4f" Nov 03 14:34:09.455410 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:34:09.455356 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:34:12.801121 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:12.801064 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf"] Nov 03 14:34:12.802503 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:12.802475 5060 cpu_manager.go:414] "containerMap: removing container" podUID="74a86163-4d09-49e1-945d-521f611bda7e" containerName="maas-api" Nov 03 14:34:12.802656 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:12.802644 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74a86163-4d09-49e1-945d-521f611bda7e" containerName="maas-api" Nov 03 14:34:12.802746 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:12.802737 5060 cpu_manager.go:414] "containerMap: removing container" podUID="599e862d-75d4-40f6-9750-806110c5d308" containerName="istio-proxy" Nov 03 14:34:12.802818 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:12.802808 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="599e862d-75d4-40f6-9750-806110c5d308" containerName="istio-proxy" Nov 03 14:34:12.802923 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:12.802913 5060 cpu_manager.go:414] "containerMap: removing container" podUID="07fa990f-370e-4d84-b7e6-340f4caa6348" containerName="istio-proxy" Nov 03 14:34:12.802992 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:12.802983 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07fa990f-370e-4d84-b7e6-340f4caa6348" containerName="istio-proxy" Nov 03 14:34:12.803077 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:12.803067 5060 cpu_manager.go:414] "containerMap: removing container" podUID="599e862d-75d4-40f6-9750-806110c5d308" containerName="istio-proxy" Nov 03 14:34:12.803146 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:12.803137 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="599e862d-75d4-40f6-9750-806110c5d308" containerName="istio-proxy" Nov 03 14:34:12.803374 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:12.803354 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="599e862d-75d4-40f6-9750-806110c5d308" containerName="istio-proxy" Nov 03 14:34:12.803374 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:12.803374 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="07fa990f-370e-4d84-b7e6-340f4caa6348" containerName="istio-proxy" Nov 03 14:34:12.803566 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:12.803393 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="599e862d-75d4-40f6-9750-806110c5d308" containerName="istio-proxy" Nov 03 14:34:12.803566 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:12.803404 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="74a86163-4d09-49e1-945d-521f611bda7e" containerName="maas-api" Nov 03 14:34:12.809325 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:12.809304 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" Nov 03 14:34:12.835513 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:12.835484 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf"] Nov 03 14:34:12.842615 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:12.842580 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-4a0349-openshift-default-dockercfg-ml2mc\"" Nov 03 14:34:12.883872 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:12.883826 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae02904-4e94-487e-a422-2749cdb51c4f-credential-socket\") pod \"e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf\" (UID: \"7ae02904-4e94-487e-a422-2749cdb51c4f\") " pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" Nov 03 14:34:12.884041 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:12.883921 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ae02904-4e94-487e-a422-2749cdb51c4f-istiod-ca-cert\") pod \"e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf\" (UID: \"7ae02904-4e94-487e-a422-2749cdb51c4f\") " pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" Nov 03 14:34:12.884041 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:12.883970 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae02904-4e94-487e-a422-2749cdb51c4f-workload-socket\") pod \"e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf\" (UID: \"7ae02904-4e94-487e-a422-2749cdb51c4f\") " pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" Nov 03 14:34:12.884041 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:12.884005 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ae02904-4e94-487e-a422-2749cdb51c4f-istio-token\") pod \"e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf\" (UID: \"7ae02904-4e94-487e-a422-2749cdb51c4f\") " pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" Nov 03 14:34:12.884174 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:12.884060 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ae02904-4e94-487e-a422-2749cdb51c4f-workload-certs\") pod \"e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf\" (UID: \"7ae02904-4e94-487e-a422-2749cdb51c4f\") " pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" Nov 03 14:34:12.884174 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:12.884149 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ae02904-4e94-487e-a422-2749cdb51c4f-istio-envoy\") pod \"e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf\" (UID: \"7ae02904-4e94-487e-a422-2749cdb51c4f\") " pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" Nov 03 14:34:12.884251 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:12.884175 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nv8h\" (UniqueName: \"kubernetes.io/projected/7ae02904-4e94-487e-a422-2749cdb51c4f-kube-api-access-7nv8h\") pod \"e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf\" (UID: \"7ae02904-4e94-487e-a422-2749cdb51c4f\") " pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" Nov 03 14:34:12.884251 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:12.884198 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ae02904-4e94-487e-a422-2749cdb51c4f-istio-podinfo\") pod \"e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf\" (UID: \"7ae02904-4e94-487e-a422-2749cdb51c4f\") " pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" Nov 03 14:34:12.884251 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:12.884221 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ae02904-4e94-487e-a422-2749cdb51c4f-istio-data\") pod \"e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf\" (UID: \"7ae02904-4e94-487e-a422-2749cdb51c4f\") " pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" Nov 03 14:34:12.990888 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:12.985662 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ae02904-4e94-487e-a422-2749cdb51c4f-istio-envoy\") pod \"e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf\" (UID: \"7ae02904-4e94-487e-a422-2749cdb51c4f\") " pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" Nov 03 14:34:12.990888 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:12.987239 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nv8h\" (UniqueName: \"kubernetes.io/projected/7ae02904-4e94-487e-a422-2749cdb51c4f-kube-api-access-7nv8h\") pod \"e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf\" (UID: \"7ae02904-4e94-487e-a422-2749cdb51c4f\") " pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" Nov 03 14:34:12.990888 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:12.989279 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ae02904-4e94-487e-a422-2749cdb51c4f-istio-podinfo\") pod \"e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf\" (UID: \"7ae02904-4e94-487e-a422-2749cdb51c4f\") " pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" Nov 03 14:34:12.990888 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:12.989322 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ae02904-4e94-487e-a422-2749cdb51c4f-istio-data\") pod \"e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf\" (UID: \"7ae02904-4e94-487e-a422-2749cdb51c4f\") " pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" Nov 03 14:34:12.990888 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:12.989438 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae02904-4e94-487e-a422-2749cdb51c4f-credential-socket\") pod \"e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf\" (UID: \"7ae02904-4e94-487e-a422-2749cdb51c4f\") " pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" Nov 03 14:34:12.990888 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:12.989504 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ae02904-4e94-487e-a422-2749cdb51c4f-istiod-ca-cert\") pod \"e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf\" (UID: \"7ae02904-4e94-487e-a422-2749cdb51c4f\") " pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" Nov 03 14:34:12.990888 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:12.989552 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae02904-4e94-487e-a422-2749cdb51c4f-workload-socket\") pod \"e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf\" (UID: \"7ae02904-4e94-487e-a422-2749cdb51c4f\") " pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" Nov 03 14:34:12.990888 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:12.989589 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ae02904-4e94-487e-a422-2749cdb51c4f-istio-token\") pod \"e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf\" (UID: \"7ae02904-4e94-487e-a422-2749cdb51c4f\") " pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" Nov 03 14:34:12.990888 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:12.989651 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ae02904-4e94-487e-a422-2749cdb51c4f-workload-certs\") pod \"e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf\" (UID: \"7ae02904-4e94-487e-a422-2749cdb51c4f\") " pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" Nov 03 14:34:12.990888 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:12.990091 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ae02904-4e94-487e-a422-2749cdb51c4f-workload-certs\") pod \"e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf\" (UID: \"7ae02904-4e94-487e-a422-2749cdb51c4f\") " pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" Nov 03 14:34:12.991557 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:12.991090 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ae02904-4e94-487e-a422-2749cdb51c4f-istiod-ca-cert\") pod \"e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf\" (UID: \"7ae02904-4e94-487e-a422-2749cdb51c4f\") " pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" Nov 03 14:34:12.991557 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:12.991324 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ae02904-4e94-487e-a422-2749cdb51c4f-istio-data\") pod \"e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf\" (UID: \"7ae02904-4e94-487e-a422-2749cdb51c4f\") " pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" Nov 03 14:34:12.991557 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:12.991541 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae02904-4e94-487e-a422-2749cdb51c4f-credential-socket\") pod \"e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf\" (UID: \"7ae02904-4e94-487e-a422-2749cdb51c4f\") " pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" Nov 03 14:34:12.991784 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:12.991764 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae02904-4e94-487e-a422-2749cdb51c4f-workload-socket\") pod \"e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf\" (UID: \"7ae02904-4e94-487e-a422-2749cdb51c4f\") " pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" Nov 03 14:34:12.995009 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:12.994982 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ae02904-4e94-487e-a422-2749cdb51c4f-istio-podinfo\") pod \"e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf\" (UID: \"7ae02904-4e94-487e-a422-2749cdb51c4f\") " pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" Nov 03 14:34:13.035860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:12.996160 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ae02904-4e94-487e-a422-2749cdb51c4f-istio-envoy\") pod \"e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf\" (UID: \"7ae02904-4e94-487e-a422-2749cdb51c4f\") " pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" Nov 03 14:34:13.035860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:13.017564 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nv8h\" (UniqueName: \"kubernetes.io/projected/7ae02904-4e94-487e-a422-2749cdb51c4f-kube-api-access-7nv8h\") pod \"e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf\" (UID: \"7ae02904-4e94-487e-a422-2749cdb51c4f\") " pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" Nov 03 14:34:13.054901 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:13.037441 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ae02904-4e94-487e-a422-2749cdb51c4f-istio-token\") pod \"e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf\" (UID: \"7ae02904-4e94-487e-a422-2749cdb51c4f\") " pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" Nov 03 14:34:13.136855 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:13.130056 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" Nov 03 14:34:13.481775 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:34:13.481746 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ae02904_4e94_487e_a422_2749cdb51c4f.slice/crio-688057c779c84db33fe71f5c0c38710f3249d0ec515615a4438385d9ac8587f7 WatchSource:0}: Error finding container 688057c779c84db33fe71f5c0c38710f3249d0ec515615a4438385d9ac8587f7: Status 404 returned error can't find the container with id 688057c779c84db33fe71f5c0c38710f3249d0ec515615a4438385d9ac8587f7 Nov 03 14:34:13.482419 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:13.482394 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf"] Nov 03 14:34:13.484228 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:13.484201 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:34:13.484311 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:13.484270 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:34:13.484311 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:13.484297 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:34:13.866587 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:13.866548 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" event={"ID":"7ae02904-4e94-487e-a422-2749cdb51c4f","Type":"ContainerStarted","Data":"083147e066d5126568ef8097c11067f2637dd8df409af907b3d6793beb16add6"} Nov 03 14:34:13.867098 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:13.866599 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" event={"ID":"7ae02904-4e94-487e-a422-2749cdb51c4f","Type":"ContainerStarted","Data":"688057c779c84db33fe71f5c0c38710f3249d0ec515615a4438385d9ac8587f7"} Nov 03 14:34:13.929092 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:13.929035 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" podStartSLOduration=1.929017467 podStartE2EDuration="1.929017467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:34:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:34:13.927983162 +0000 UTC m=+1387.816772743" watchObservedRunningTime="2026-11-03 14:34:13.929017467 +0000 UTC m=+1387.817807050" Nov 03 14:34:14.130967 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:14.130871 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" Nov 03 14:34:14.136851 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:14.136808 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" Nov 03 14:34:14.873258 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:14.873222 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" Nov 03 14:34:14.874288 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:14.874270 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" Nov 03 14:34:19.110475 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:19.110436 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6"] Nov 03 14:34:19.114657 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:19.114640 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6" Nov 03 14:34:19.148188 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:19.148135 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:34:19.148371 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:19.148233 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 14:34:19.148371 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:19.148341 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7cpjn\"" Nov 03 14:34:19.148371 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:19.148368 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-cb587c-serving-cert\"" Nov 03 14:34:19.148556 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:19.148448 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:34:19.149369 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:19.149340 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6"] Nov 03 14:34:19.183246 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:19.183216 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b93ca36c-db33-4a19-b384-331987d1bb9f-metrics-certs\") pod \"maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6\" (UID: \"b93ca36c-db33-4a19-b384-331987d1bb9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6" Nov 03 14:34:19.183418 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:19.183266 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b93ca36c-db33-4a19-b384-331987d1bb9f-openshift-service-ca\") pod \"maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6\" (UID: \"b93ca36c-db33-4a19-b384-331987d1bb9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6" Nov 03 14:34:19.183418 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:19.183347 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvvpn\" (UniqueName: \"kubernetes.io/projected/b93ca36c-db33-4a19-b384-331987d1bb9f-kube-api-access-zvvpn\") pod \"maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6\" (UID: \"b93ca36c-db33-4a19-b384-331987d1bb9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6" Nov 03 14:34:19.183418 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:19.183412 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b93ca36c-db33-4a19-b384-331987d1bb9f-maas-api-tls\") pod \"maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6\" (UID: \"b93ca36c-db33-4a19-b384-331987d1bb9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6" Nov 03 14:34:19.284602 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:19.284566 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b93ca36c-db33-4a19-b384-331987d1bb9f-metrics-certs\") pod \"maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6\" (UID: \"b93ca36c-db33-4a19-b384-331987d1bb9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6" Nov 03 14:34:19.284779 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:19.284617 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b93ca36c-db33-4a19-b384-331987d1bb9f-openshift-service-ca\") pod \"maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6\" (UID: \"b93ca36c-db33-4a19-b384-331987d1bb9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6" Nov 03 14:34:19.284779 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:19.284640 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvvpn\" (UniqueName: \"kubernetes.io/projected/b93ca36c-db33-4a19-b384-331987d1bb9f-kube-api-access-zvvpn\") pod \"maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6\" (UID: \"b93ca36c-db33-4a19-b384-331987d1bb9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6" Nov 03 14:34:19.284779 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:19.284666 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b93ca36c-db33-4a19-b384-331987d1bb9f-maas-api-tls\") pod \"maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6\" (UID: \"b93ca36c-db33-4a19-b384-331987d1bb9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6" Nov 03 14:34:19.284943 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:34:19.284814 5060 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-cb587c-serving-cert: secret "maas-api-e2e-api-a-cb587c-serving-cert" not found Nov 03 14:34:19.284943 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:34:19.284932 5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b93ca36c-db33-4a19-b384-331987d1bb9f-maas-api-tls podName:b93ca36c-db33-4a19-b384-331987d1bb9f nodeName:}" failed. No retries permitted until 2026-11-03 14:34:19.784909941 +0000 UTC m=+1393.673699520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b93ca36c-db33-4a19-b384-331987d1bb9f-maas-api-tls") pod "maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6" (UID: "b93ca36c-db33-4a19-b384-331987d1bb9f") : secret "maas-api-e2e-api-a-cb587c-serving-cert" not found Nov 03 14:34:19.285359 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:19.285328 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b93ca36c-db33-4a19-b384-331987d1bb9f-openshift-service-ca\") pod \"maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6\" (UID: \"b93ca36c-db33-4a19-b384-331987d1bb9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6" Nov 03 14:34:19.287118 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:19.287098 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b93ca36c-db33-4a19-b384-331987d1bb9f-metrics-certs\") pod \"maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6\" (UID: \"b93ca36c-db33-4a19-b384-331987d1bb9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6" Nov 03 14:34:19.334620 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:19.333455 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvvpn\" (UniqueName: \"kubernetes.io/projected/b93ca36c-db33-4a19-b384-331987d1bb9f-kube-api-access-zvvpn\") pod \"maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6\" (UID: \"b93ca36c-db33-4a19-b384-331987d1bb9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6" Nov 03 14:34:19.791405 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:19.791369 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b93ca36c-db33-4a19-b384-331987d1bb9f-maas-api-tls\") pod \"maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6\" (UID: \"b93ca36c-db33-4a19-b384-331987d1bb9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6" Nov 03 14:34:19.793699 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:19.793673 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b93ca36c-db33-4a19-b384-331987d1bb9f-maas-api-tls\") pod \"maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6\" (UID: \"b93ca36c-db33-4a19-b384-331987d1bb9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6" Nov 03 14:34:20.030626 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:20.030581 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6" Nov 03 14:34:20.365125 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:20.365095 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6"] Nov 03 14:34:20.366121 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:34:20.366093 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb93ca36c_db33_4a19_b384_331987d1bb9f.slice/crio-c3b40240082fb10e0562f2b8b4e4bb3270847431c5bbe3f1970a5245845761e9 WatchSource:0}: Error finding container c3b40240082fb10e0562f2b8b4e4bb3270847431c5bbe3f1970a5245845761e9: Status 404 returned error can't find the container with id c3b40240082fb10e0562f2b8b4e4bb3270847431c5bbe3f1970a5245845761e9 Nov 03 14:34:20.944859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:20.940564 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6" event={"ID":"b93ca36c-db33-4a19-b384-331987d1bb9f","Type":"ContainerStarted","Data":"c3b40240082fb10e0562f2b8b4e4bb3270847431c5bbe3f1970a5245845761e9"} Nov 03 14:34:21.349859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:21.348853 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7"] Nov 03 14:34:21.369935 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:21.369857 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" Nov 03 14:34:21.498865 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:21.497010 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43465f0f-764a-44c5-b762-9b6932953b41-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5h7r7\" (UID: \"43465f0f-764a-44c5-b762-9b6932953b41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" Nov 03 14:34:21.498865 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:21.497104 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43465f0f-764a-44c5-b762-9b6932953b41-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5h7r7\" (UID: \"43465f0f-764a-44c5-b762-9b6932953b41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" Nov 03 14:34:21.498865 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:21.497219 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43465f0f-764a-44c5-b762-9b6932953b41-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5h7r7\" (UID: \"43465f0f-764a-44c5-b762-9b6932953b41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" Nov 03 14:34:21.498865 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:21.497296 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43465f0f-764a-44c5-b762-9b6932953b41-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5h7r7\" (UID: \"43465f0f-764a-44c5-b762-9b6932953b41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" Nov 03 14:34:21.498865 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:21.497326 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjcsn\" (UniqueName: \"kubernetes.io/projected/43465f0f-764a-44c5-b762-9b6932953b41-kube-api-access-jjcsn\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5h7r7\" (UID: \"43465f0f-764a-44c5-b762-9b6932953b41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" Nov 03 14:34:21.498865 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:21.497410 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43465f0f-764a-44c5-b762-9b6932953b41-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5h7r7\" (UID: \"43465f0f-764a-44c5-b762-9b6932953b41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" Nov 03 14:34:21.498865 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:21.497484 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43465f0f-764a-44c5-b762-9b6932953b41-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5h7r7\" (UID: \"43465f0f-764a-44c5-b762-9b6932953b41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" Nov 03 14:34:21.498865 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:21.497521 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43465f0f-764a-44c5-b762-9b6932953b41-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5h7r7\" (UID: \"43465f0f-764a-44c5-b762-9b6932953b41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" Nov 03 14:34:21.498865 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:21.497581 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43465f0f-764a-44c5-b762-9b6932953b41-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5h7r7\" (UID: \"43465f0f-764a-44c5-b762-9b6932953b41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" Nov 03 14:34:21.620914 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:21.592986 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7"] Nov 03 14:34:21.622202 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:21.622164 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43465f0f-764a-44c5-b762-9b6932953b41-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5h7r7\" (UID: \"43465f0f-764a-44c5-b762-9b6932953b41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" Nov 03 14:34:21.622430 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:21.622413 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43465f0f-764a-44c5-b762-9b6932953b41-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5h7r7\" (UID: \"43465f0f-764a-44c5-b762-9b6932953b41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" Nov 03 14:34:21.622551 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:21.622534 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43465f0f-764a-44c5-b762-9b6932953b41-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5h7r7\" (UID: \"43465f0f-764a-44c5-b762-9b6932953b41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" Nov 03 14:34:21.622703 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:21.622689 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43465f0f-764a-44c5-b762-9b6932953b41-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5h7r7\" (UID: \"43465f0f-764a-44c5-b762-9b6932953b41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" Nov 03 14:34:21.629891 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:21.622856 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43465f0f-764a-44c5-b762-9b6932953b41-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5h7r7\" (UID: \"43465f0f-764a-44c5-b762-9b6932953b41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" Nov 03 14:34:21.630156 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:21.630133 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43465f0f-764a-44c5-b762-9b6932953b41-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5h7r7\" (UID: \"43465f0f-764a-44c5-b762-9b6932953b41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" Nov 03 14:34:21.630273 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:21.630258 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjcsn\" (UniqueName: \"kubernetes.io/projected/43465f0f-764a-44c5-b762-9b6932953b41-kube-api-access-jjcsn\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5h7r7\" (UID: \"43465f0f-764a-44c5-b762-9b6932953b41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" Nov 03 14:34:21.640881 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:21.630438 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43465f0f-764a-44c5-b762-9b6932953b41-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5h7r7\" (UID: \"43465f0f-764a-44c5-b762-9b6932953b41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" Nov 03 14:34:21.641169 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:21.641143 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43465f0f-764a-44c5-b762-9b6932953b41-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5h7r7\" (UID: \"43465f0f-764a-44c5-b762-9b6932953b41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" Nov 03 14:34:21.642777 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:21.642752 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43465f0f-764a-44c5-b762-9b6932953b41-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5h7r7\" (UID: \"43465f0f-764a-44c5-b762-9b6932953b41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" Nov 03 14:34:21.643217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:21.643195 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43465f0f-764a-44c5-b762-9b6932953b41-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5h7r7\" (UID: \"43465f0f-764a-44c5-b762-9b6932953b41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" Nov 03 14:34:21.645720 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:21.644301 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43465f0f-764a-44c5-b762-9b6932953b41-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5h7r7\" (UID: \"43465f0f-764a-44c5-b762-9b6932953b41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" Nov 03 14:34:21.645720 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:21.644561 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43465f0f-764a-44c5-b762-9b6932953b41-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5h7r7\" (UID: \"43465f0f-764a-44c5-b762-9b6932953b41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" Nov 03 14:34:21.645720 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:21.645231 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43465f0f-764a-44c5-b762-9b6932953b41-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5h7r7\" (UID: \"43465f0f-764a-44c5-b762-9b6932953b41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" Nov 03 14:34:21.809881 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:21.803735 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjcsn\" (UniqueName: \"kubernetes.io/projected/43465f0f-764a-44c5-b762-9b6932953b41-kube-api-access-jjcsn\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5h7r7\" (UID: \"43465f0f-764a-44c5-b762-9b6932953b41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" Nov 03 14:34:21.847856 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:21.836227 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43465f0f-764a-44c5-b762-9b6932953b41-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5h7r7\" (UID: \"43465f0f-764a-44c5-b762-9b6932953b41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" Nov 03 14:34:21.873113 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:21.865529 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43465f0f-764a-44c5-b762-9b6932953b41-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5h7r7\" (UID: \"43465f0f-764a-44c5-b762-9b6932953b41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" Nov 03 14:34:21.886031 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:21.885955 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43465f0f-764a-44c5-b762-9b6932953b41-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5h7r7\" (UID: \"43465f0f-764a-44c5-b762-9b6932953b41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" Nov 03 14:34:21.974328 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:21.974286 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6" event={"ID":"b93ca36c-db33-4a19-b384-331987d1bb9f","Type":"ContainerStarted","Data":"b446a8eec756fc59ff8636739e52bf8f77a36d0f4a5d20c9c08e59316cea5ad5"} Nov 03 14:34:21.988439 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:21.988312 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6" Nov 03 14:34:22.143099 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:22.143010 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" Nov 03 14:34:22.584563 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:22.584511 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6" podStartSLOduration=3.14130554 podStartE2EDuration="3.584495674s" totalImagesPullingTime="443.190134ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:34:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:34:22.06823974 +0000 UTC m=+1395.957029324" watchObservedRunningTime="2026-11-03 14:34:22.584495674 +0000 UTC m=+1396.473285281" Nov 03 14:34:22.587962 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:34:22.587937 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43465f0f_764a_44c5_b762_9b6932953b41.slice/crio-0cb0aacb1086e7062da8a6481f750797cf3443a9abe14a9b70011b78d8f1ff14 WatchSource:0}: Error finding container 0cb0aacb1086e7062da8a6481f750797cf3443a9abe14a9b70011b78d8f1ff14: Status 404 returned error can't find the container with id 0cb0aacb1086e7062da8a6481f750797cf3443a9abe14a9b70011b78d8f1ff14 Nov 03 14:34:22.589116 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:22.589096 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7"] Nov 03 14:34:22.590812 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:22.590782 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:34:22.590891 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:22.590859 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:34:22.590931 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:22.590890 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:34:22.983963 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:22.983873 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" event={"ID":"43465f0f-764a-44c5-b762-9b6932953b41","Type":"ContainerStarted","Data":"2eebb02b02b15cd18301087ba05ebb49a583321cc83f4dd05a702c9591ad8226"} Nov 03 14:34:22.983963 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:22.983935 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" event={"ID":"43465f0f-764a-44c5-b762-9b6932953b41","Type":"ContainerStarted","Data":"0cb0aacb1086e7062da8a6481f750797cf3443a9abe14a9b70011b78d8f1ff14"} Nov 03 14:34:23.062472 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:23.062404 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" podStartSLOduration=2.06238203 podStartE2EDuration="2.06238203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:34:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:34:23.048314594 +0000 UTC m=+1396.937104176" watchObservedRunningTime="2026-11-03 14:34:23.06238203 +0000 UTC m=+1396.951171613" Nov 03 14:34:23.144471 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:23.144393 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" Nov 03 14:34:23.145981 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:23.145949 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 14:34:23.146142 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:23.146010 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" podUID="43465f0f-764a-44c5-b762-9b6932953b41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 14:34:24.147745 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:24.147698 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 14:34:24.148303 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:24.148148 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" podUID="43465f0f-764a-44c5-b762-9b6932953b41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 14:34:24.519748 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:24.519697 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7322-68ae22"] Nov 03 14:34:24.526537 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:24.525869 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-68ae22" Nov 03 14:34:24.552030 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:24.551194 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-9sm86\"" Nov 03 14:34:24.561558 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:24.561527 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7322-68ae22"] Nov 03 14:34:24.693176 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:24.693136 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm4wr\" (UniqueName: \"kubernetes.io/projected/f2717800-891c-4466-82d9-c6fb167be30b-kube-api-access-dm4wr\") pod \"test-curl-7322-68ae22\" (UID: \"f2717800-891c-4466-82d9-c6fb167be30b\") " pod="opendatahub/test-curl-7322-68ae22" Nov 03 14:34:24.794657 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:24.794576 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm4wr\" (UniqueName: \"kubernetes.io/projected/f2717800-891c-4466-82d9-c6fb167be30b-kube-api-access-dm4wr\") pod \"test-curl-7322-68ae22\" (UID: \"f2717800-891c-4466-82d9-c6fb167be30b\") " pod="opendatahub/test-curl-7322-68ae22" Nov 03 14:34:24.840729 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:24.840694 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm4wr\" (UniqueName: \"kubernetes.io/projected/f2717800-891c-4466-82d9-c6fb167be30b-kube-api-access-dm4wr\") pod \"test-curl-7322-68ae22\" (UID: \"f2717800-891c-4466-82d9-c6fb167be30b\") " pod="opendatahub/test-curl-7322-68ae22" Nov 03 14:34:24.869387 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:24.869057 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-68ae22" Nov 03 14:34:25.144189 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:25.144096 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 14:34:25.144189 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:25.144157 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" podUID="43465f0f-764a-44c5-b762-9b6932953b41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 14:34:25.227961 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:25.227907 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7322-68ae22"] Nov 03 14:34:26.030659 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:26.030623 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7322-68ae22" event={"ID":"f2717800-891c-4466-82d9-c6fb167be30b","Type":"ContainerStarted","Data":"f7524dbc6feb645728a4efe885f20d9fe7244b20fec00886d603a93c4a894a6f"} Nov 03 14:34:26.030871 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:26.030667 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7322-68ae22" event={"ID":"f2717800-891c-4466-82d9-c6fb167be30b","Type":"ContainerStarted","Data":"ef7ad92e95de2731255a1d22e2e8ea9d773b16aaa8dec694f5d3548bca4547c3"} Nov 03 14:34:26.077332 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:26.077280 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7322-68ae22" podStartSLOduration=2.077266329 podStartE2EDuration="2.077266329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:34:26.07492453 +0000 UTC m=+1399.963714110" watchObservedRunningTime="2026-11-03 14:34:26.077266329 +0000 UTC m=+1399.966055909" Nov 03 14:34:26.144440 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:26.144382 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 14:34:26.144440 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:26.144435 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" podUID="43465f0f-764a-44c5-b762-9b6932953b41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 14:34:27.038244 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:27.038213 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7322-68ae22"] Nov 03 14:34:27.042028 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:27.041997 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7322-68ae22"] Nov 03 14:34:27.144449 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:27.144414 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 14:34:27.144657 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:27.144482 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" podUID="43465f0f-764a-44c5-b762-9b6932953b41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 14:34:27.225735 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:27.225701 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7322-9e30f2"] Nov 03 14:34:27.226275 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:27.226259 5060 cpu_manager.go:414] "containerMap: removing container" podUID="f2717800-891c-4466-82d9-c6fb167be30b" containerName="test-curl-7322-68ae22" Nov 03 14:34:27.226325 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:27.226277 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2717800-891c-4466-82d9-c6fb167be30b" containerName="test-curl-7322-68ae22" Nov 03 14:34:27.226394 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:27.226385 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2717800-891c-4466-82d9-c6fb167be30b" containerName="test-curl-7322-68ae22" Nov 03 14:34:27.229986 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:27.229969 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-9e30f2" Nov 03 14:34:27.241029 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:27.241007 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7322-9e30f2"] Nov 03 14:34:27.252990 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:27.252960 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnn45\" (UniqueName: \"kubernetes.io/projected/dd87133e-e567-4156-97d2-c32561ca7f4d-kube-api-access-tnn45\") pod \"test-curl-7322-9e30f2\" (UID: \"dd87133e-e567-4156-97d2-c32561ca7f4d\") " pod="opendatahub/test-curl-7322-9e30f2" Nov 03 14:34:27.354591 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:27.354515 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnn45\" (UniqueName: \"kubernetes.io/projected/dd87133e-e567-4156-97d2-c32561ca7f4d-kube-api-access-tnn45\") pod \"test-curl-7322-9e30f2\" (UID: \"dd87133e-e567-4156-97d2-c32561ca7f4d\") " pod="opendatahub/test-curl-7322-9e30f2" Nov 03 14:34:27.379666 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:27.379639 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnn45\" (UniqueName: \"kubernetes.io/projected/dd87133e-e567-4156-97d2-c32561ca7f4d-kube-api-access-tnn45\") pod \"test-curl-7322-9e30f2\" (UID: \"dd87133e-e567-4156-97d2-c32561ca7f4d\") " pod="opendatahub/test-curl-7322-9e30f2" Nov 03 14:34:27.543086 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:27.543054 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-9e30f2" Nov 03 14:34:27.856469 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:27.856431 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7322-9e30f2"] Nov 03 14:34:27.858095 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:34:27.858061 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd87133e_e567_4156_97d2_c32561ca7f4d.slice/crio-dcfef0441df634110fd1b7e88cdb3429ca609b2516d8c8ae86bf6a308b51577d WatchSource:0}: Error finding container dcfef0441df634110fd1b7e88cdb3429ca609b2516d8c8ae86bf6a308b51577d: Status 404 returned error can't find the container with id dcfef0441df634110fd1b7e88cdb3429ca609b2516d8c8ae86bf6a308b51577d Nov 03 14:34:28.047193 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:28.047152 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7322-9e30f2" event={"ID":"dd87133e-e567-4156-97d2-c32561ca7f4d","Type":"ContainerStarted","Data":"3be0ca24d9fc221c9a13ed096de377f0fed1d4f80155d15bf90ce9b6bd4b5f42"} Nov 03 14:34:28.047679 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:28.047207 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7322-9e30f2" event={"ID":"dd87133e-e567-4156-97d2-c32561ca7f4d","Type":"ContainerStarted","Data":"dcfef0441df634110fd1b7e88cdb3429ca609b2516d8c8ae86bf6a308b51577d"} Nov 03 14:34:28.047679 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:28.047307 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7322-68ae22" podUID="f2717800-891c-4466-82d9-c6fb167be30b" containerName="test-curl-7322-68ae22" containerID="cri-o://f7524dbc6feb645728a4efe885f20d9fe7244b20fec00886d603a93c4a894a6f" gracePeriod=2 Nov 03 14:34:28.091908 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:28.091853 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7322-9e30f2" podStartSLOduration=1.091828403 podStartE2EDuration="1.091828403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:34:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:34:28.089226735 +0000 UTC m=+1401.978016315" watchObservedRunningTime="2026-11-03 14:34:28.091828403 +0000 UTC m=+1401.980617984" Nov 03 14:34:28.113857 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:34:28.107039 5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7322-68ae22\" is forbidden: User \"system:node:ip-10-0-143-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-171.ec2.internal' and this object" podUID="f2717800-891c-4466-82d9-c6fb167be30b" pod="opendatahub/test-curl-7322-68ae22" Nov 03 14:34:28.123200 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:34:28.123083 5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7322-68ae22\" is forbidden: User \"system:node:ip-10-0-143-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-171.ec2.internal' and this object" podUID="f2717800-891c-4466-82d9-c6fb167be30b" pod="opendatahub/test-curl-7322-68ae22" Nov 03 14:34:28.144597 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:28.144571 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 14:34:28.144754 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:28.144629 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" podUID="43465f0f-764a-44c5-b762-9b6932953b41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 14:34:28.996938 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:28.996909 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6" Nov 03 14:34:29.032322 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:29.032291 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7322-9e30f2"] Nov 03 14:34:29.038184 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:29.038142 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7322-9e30f2"] Nov 03 14:34:29.144780 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:29.144725 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 14:34:29.145282 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:29.144847 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" podUID="43465f0f-764a-44c5-b762-9b6932953b41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 14:34:30.066071 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:30.066039 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7322-9e30f2" podUID="dd87133e-e567-4156-97d2-c32561ca7f4d" containerName="test-curl-7322-9e30f2" containerID="cri-o://3be0ca24d9fc221c9a13ed096de377f0fed1d4f80155d15bf90ce9b6bd4b5f42" gracePeriod=2 Nov 03 14:34:30.080258 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:34:30.080222 5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7322-9e30f2\" is forbidden: User \"system:node:ip-10-0-143-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-171.ec2.internal' and this object" podUID="dd87133e-e567-4156-97d2-c32561ca7f4d" pod="opendatahub/test-curl-7322-9e30f2" Nov 03 14:34:30.144772 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:30.144729 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 14:34:30.144961 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:30.144781 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" podUID="43465f0f-764a-44c5-b762-9b6932953b41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 14:34:30.272482 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:30.272457 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-68ae22" Nov 03 14:34:30.290182 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:30.290141 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2717800-891c-4466-82d9-c6fb167be30b-kube-api-access-dm4wr\" (UniqueName: \"kubernetes.io/projected/f2717800-891c-4466-82d9-c6fb167be30b-kube-api-access-dm4wr\") pod \"f2717800-891c-4466-82d9-c6fb167be30b\" (UID: \"f2717800-891c-4466-82d9-c6fb167be30b\") " Nov 03 14:34:30.292183 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:30.292161 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2717800-891c-4466-82d9-c6fb167be30b-kube-api-access-dm4wr" pod "f2717800-891c-4466-82d9-c6fb167be30b" (UID: "f2717800-891c-4466-82d9-c6fb167be30b"). InnerVolumeSpecName "kube-api-access-dm4wr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:34:30.392038 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:30.391941 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm4wr\" (UniqueName: \"kubernetes.io/projected/f2717800-891c-4466-82d9-c6fb167be30b-kube-api-access-dm4wr\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:34:31.078885 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:31.078850 5060 generic.go:356] "Generic (PLEG): container finished" podID="f2717800-891c-4466-82d9-c6fb167be30b" containerID="f7524dbc6feb645728a4efe885f20d9fe7244b20fec00886d603a93c4a894a6f" exitCode=137 Nov 03 14:34:31.079102 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:31.078971 5060 scope.go:122] "RemoveContainer" containerID="f7524dbc6feb645728a4efe885f20d9fe7244b20fec00886d603a93c4a894a6f" Nov 03 14:34:31.079169 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:31.079111 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-68ae22" Nov 03 14:34:31.106554 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:31.104991 5060 scope.go:122] "RemoveContainer" containerID="f7524dbc6feb645728a4efe885f20d9fe7244b20fec00886d603a93c4a894a6f" Nov 03 14:34:31.106554 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:34:31.106279 5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7322-68ae22\" is forbidden: User \"system:node:ip-10-0-143-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-171.ec2.internal' and this object" podUID="f2717800-891c-4466-82d9-c6fb167be30b" pod="opendatahub/test-curl-7322-68ae22" Nov 03 14:34:31.107029 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:34:31.106904 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7524dbc6feb645728a4efe885f20d9fe7244b20fec00886d603a93c4a894a6f\": container with ID starting with f7524dbc6feb645728a4efe885f20d9fe7244b20fec00886d603a93c4a894a6f not found: ID does not exist" containerID="f7524dbc6feb645728a4efe885f20d9fe7244b20fec00886d603a93c4a894a6f" Nov 03 14:34:31.107029 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:31.106958 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7524dbc6feb645728a4efe885f20d9fe7244b20fec00886d603a93c4a894a6f"} err="failed to get container status \"f7524dbc6feb645728a4efe885f20d9fe7244b20fec00886d603a93c4a894a6f\": rpc error: code = NotFound desc = could not find container \"f7524dbc6feb645728a4efe885f20d9fe7244b20fec00886d603a93c4a894a6f\": container with ID starting with f7524dbc6feb645728a4efe885f20d9fe7244b20fec00886d603a93c4a894a6f not found: ID does not exist" Nov 03 14:34:31.144153 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:31.144122 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 14:34:31.144311 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:31.144173 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" podUID="43465f0f-764a-44c5-b762-9b6932953b41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 14:34:32.260600 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:32.260501 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2717800-891c-4466-82d9-c6fb167be30b" path="/var/lib/kubelet/pods/f2717800-891c-4466-82d9-c6fb167be30b/volumes" Nov 03 14:34:32.359682 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:32.359554 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-9e30f2" Nov 03 14:34:32.426236 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:32.426206 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd87133e-e567-4156-97d2-c32561ca7f4d-kube-api-access-tnn45\" (UniqueName: \"kubernetes.io/projected/dd87133e-e567-4156-97d2-c32561ca7f4d-kube-api-access-tnn45\") pod \"dd87133e-e567-4156-97d2-c32561ca7f4d\" (UID: \"dd87133e-e567-4156-97d2-c32561ca7f4d\") " Nov 03 14:34:32.428460 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:32.428430 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd87133e-e567-4156-97d2-c32561ca7f4d-kube-api-access-tnn45" pod "dd87133e-e567-4156-97d2-c32561ca7f4d" (UID: "dd87133e-e567-4156-97d2-c32561ca7f4d"). InnerVolumeSpecName "kube-api-access-tnn45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:34:32.527633 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:32.527587 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnn45\" (UniqueName: \"kubernetes.io/projected/dd87133e-e567-4156-97d2-c32561ca7f4d-kube-api-access-tnn45\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:34:32.534584 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:32.534547 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" Nov 03 14:34:32.534994 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:32.534971 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" Nov 03 14:34:32.536046 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:32.536025 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" Nov 03 14:34:33.095157 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:33.095124 5060 generic.go:356] "Generic (PLEG): container finished" podID="dd87133e-e567-4156-97d2-c32561ca7f4d" containerID="3be0ca24d9fc221c9a13ed096de377f0fed1d4f80155d15bf90ce9b6bd4b5f42" exitCode=137 Nov 03 14:34:33.095347 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:33.095183 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-9e30f2" Nov 03 14:34:33.095347 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:33.095219 5060 scope.go:122] "RemoveContainer" containerID="3be0ca24d9fc221c9a13ed096de377f0fed1d4f80155d15bf90ce9b6bd4b5f42" Nov 03 14:34:33.107550 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:33.107526 5060 scope.go:122] "RemoveContainer" containerID="3be0ca24d9fc221c9a13ed096de377f0fed1d4f80155d15bf90ce9b6bd4b5f42" Nov 03 14:34:33.107941 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:34:33.107912 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3be0ca24d9fc221c9a13ed096de377f0fed1d4f80155d15bf90ce9b6bd4b5f42\": container with ID starting with 3be0ca24d9fc221c9a13ed096de377f0fed1d4f80155d15bf90ce9b6bd4b5f42 not found: ID does not exist" containerID="3be0ca24d9fc221c9a13ed096de377f0fed1d4f80155d15bf90ce9b6bd4b5f42" Nov 03 14:34:33.108052 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:33.107955 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3be0ca24d9fc221c9a13ed096de377f0fed1d4f80155d15bf90ce9b6bd4b5f42"} err="failed to get container status \"3be0ca24d9fc221c9a13ed096de377f0fed1d4f80155d15bf90ce9b6bd4b5f42\": rpc error: code = NotFound desc = could not find container \"3be0ca24d9fc221c9a13ed096de377f0fed1d4f80155d15bf90ce9b6bd4b5f42\": container with ID starting with 3be0ca24d9fc221c9a13ed096de377f0fed1d4f80155d15bf90ce9b6bd4b5f42 not found: ID does not exist" Nov 03 14:34:33.118823 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:34:33.118786 5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7322-9e30f2\" is forbidden: User \"system:node:ip-10-0-143-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-171.ec2.internal' and this object" podUID="dd87133e-e567-4156-97d2-c32561ca7f4d" pod="opendatahub/test-curl-7322-9e30f2" Nov 03 14:34:34.250231 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:34.250194 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd87133e-e567-4156-97d2-c32561ca7f4d" path="/var/lib/kubelet/pods/dd87133e-e567-4156-97d2-c32561ca7f4d/volumes" Nov 03 14:34:36.048505 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.048471 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce62b71d-mwjk2"] Nov 03 14:34:36.059913 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.059882 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce62b71d-mwjk2"] Nov 03 14:34:36.259460 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.259428 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1b4c668-de56-4253-9f6b-bce9b7a41cfc" path="/var/lib/kubelet/pods/b1b4c668-de56-4253-9f6b-bce9b7a41cfc/volumes" Nov 03 14:34:36.417723 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.417632 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x"] Nov 03 14:34:36.418759 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.418730 5060 cpu_manager.go:414] "containerMap: removing container" podUID="dd87133e-e567-4156-97d2-c32561ca7f4d" containerName="test-curl-7322-9e30f2" Nov 03 14:34:36.418914 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.418898 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd87133e-e567-4156-97d2-c32561ca7f4d" containerName="test-curl-7322-9e30f2" Nov 03 14:34:36.419267 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.419249 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd87133e-e567-4156-97d2-c32561ca7f4d" containerName="test-curl-7322-9e30f2" Nov 03 14:34:36.434126 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.434088 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" Nov 03 14:34:36.435224 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.435182 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z"] Nov 03 14:34:36.440048 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.440023 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x"] Nov 03 14:34:36.440242 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.440135 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" Nov 03 14:34:36.452759 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.452730 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z"] Nov 03 14:34:36.485035 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.485007 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wcq9z\" (UID: \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" Nov 03 14:34:36.485182 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.485055 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wcq9z\" (UID: \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" Nov 03 14:34:36.485182 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.485108 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rr46x\" (UID: \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" Nov 03 14:34:36.485182 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.485126 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2b4r\" (UniqueName: \"kubernetes.io/projected/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-kube-api-access-q2b4r\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rr46x\" (UID: \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" Nov 03 14:34:36.485182 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.485153 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl8sp\" (UniqueName: \"kubernetes.io/projected/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-kube-api-access-rl8sp\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wcq9z\" (UID: \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" Nov 03 14:34:36.485182 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.485171 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rr46x\" (UID: \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" Nov 03 14:34:36.485477 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.485196 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wcq9z\" (UID: \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" Nov 03 14:34:36.485477 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.485242 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rr46x\" (UID: \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" Nov 03 14:34:36.485477 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.485270 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wcq9z\" (UID: \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" Nov 03 14:34:36.485477 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.485313 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rr46x\" (UID: \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" Nov 03 14:34:36.485477 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.485351 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rr46x\" (UID: \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" Nov 03 14:34:36.485477 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.485380 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wcq9z\" (UID: \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" Nov 03 14:34:36.485477 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.485404 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rr46x\" (UID: \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" Nov 03 14:34:36.485477 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.485441 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wcq9z\" (UID: \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" Nov 03 14:34:36.485477 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.485462 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wcq9z\" (UID: \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" Nov 03 14:34:36.486565 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.485485 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wcq9z\" (UID: \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" Nov 03 14:34:36.486565 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.485505 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rr46x\" (UID: \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" Nov 03 14:34:36.486565 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.485526 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rr46x\" (UID: \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" Nov 03 14:34:36.587029 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.586998 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rr46x\" (UID: \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" Nov 03 14:34:36.587228 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.587037 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wcq9z\" (UID: \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" Nov 03 14:34:36.587228 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.587160 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rr46x\" (UID: \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" Nov 03 14:34:36.587228 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.587191 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rr46x\" (UID: \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" Nov 03 14:34:36.587228 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.587219 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wcq9z\" (UID: \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" Nov 03 14:34:36.587453 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.587243 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rr46x\" (UID: \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" Nov 03 14:34:36.587453 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.587391 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wcq9z\" (UID: \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" Nov 03 14:34:36.587453 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.587445 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wcq9z\" (UID: \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" Nov 03 14:34:36.587617 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.587483 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wcq9z\" (UID: \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" Nov 03 14:34:36.587617 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.587508 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rr46x\" (UID: \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" Nov 03 14:34:36.587617 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.587509 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rr46x\" (UID: \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" Nov 03 14:34:36.587617 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.587521 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wcq9z\" (UID: \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" Nov 03 14:34:36.587617 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.587554 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rr46x\" (UID: \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" Nov 03 14:34:36.587617 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.587576 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rr46x\" (UID: \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" Nov 03 14:34:36.587617 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.587560 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rr46x\" (UID: \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" Nov 03 14:34:36.588070 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.587626 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wcq9z\" (UID: \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" Nov 03 14:34:36.588070 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.587678 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wcq9z\" (UID: \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" Nov 03 14:34:36.588070 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.587756 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rr46x\" (UID: \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" Nov 03 14:34:36.588070 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.587782 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2b4r\" (UniqueName: \"kubernetes.io/projected/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-kube-api-access-q2b4r\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rr46x\" (UID: \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" Nov 03 14:34:36.588070 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.587812 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wcq9z\" (UID: \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" Nov 03 14:34:36.588070 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.587824 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl8sp\" (UniqueName: \"kubernetes.io/projected/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-kube-api-access-rl8sp\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wcq9z\" (UID: \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" Nov 03 14:34:36.588070 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.587865 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rr46x\" (UID: \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" Nov 03 14:34:36.588070 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.587896 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rr46x\" (UID: \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" Nov 03 14:34:36.588070 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.587900 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wcq9z\" (UID: \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" Nov 03 14:34:36.588070 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.587933 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wcq9z\" (UID: \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" Nov 03 14:34:36.588460 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.588106 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wcq9z\" (UID: \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" Nov 03 14:34:36.588460 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.588308 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wcq9z\" (UID: \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" Nov 03 14:34:36.588460 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.588407 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rr46x\" (UID: \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" Nov 03 14:34:36.589667 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.589646 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wcq9z\" (UID: \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" Nov 03 14:34:36.590190 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.590172 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rr46x\" (UID: \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" Nov 03 14:34:36.590616 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.590600 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rr46x\" (UID: \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" Nov 03 14:34:36.590847 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.590818 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wcq9z\" (UID: \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" Nov 03 14:34:36.633215 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.633128 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2b4r\" (UniqueName: \"kubernetes.io/projected/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-kube-api-access-q2b4r\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rr46x\" (UID: \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" Nov 03 14:34:36.636083 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.636049 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl8sp\" (UniqueName: \"kubernetes.io/projected/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-kube-api-access-rl8sp\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wcq9z\" (UID: \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" Nov 03 14:34:36.636260 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.636226 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-rr46x\" (UID: \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" Nov 03 14:34:36.636479 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.636460 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wcq9z\" (UID: \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" Nov 03 14:34:36.767383 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.767346 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" Nov 03 14:34:36.774712 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:36.774677 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" Nov 03 14:34:37.159409 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:37.159377 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z"] Nov 03 14:34:37.160607 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:34:37.160571 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d7d5e9c_625a_42af_98d7_52b9a8b5745e.slice/crio-07ee22ae1936865181729d4520c439765f167be06ce2bc9075acf98411ad6022 WatchSource:0}: Error finding container 07ee22ae1936865181729d4520c439765f167be06ce2bc9075acf98411ad6022: Status 404 returned error can't find the container with id 07ee22ae1936865181729d4520c439765f167be06ce2bc9075acf98411ad6022 Nov 03 14:34:37.161346 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:34:37.161320 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50a3ab6a_5b07_4d38_b00b_9b77b506ec20.slice/crio-ab8dd4cd68812eface9e2ebd48b614f5a3255311e8be3ec879eff6e721653630 WatchSource:0}: Error finding container ab8dd4cd68812eface9e2ebd48b614f5a3255311e8be3ec879eff6e721653630: Status 404 returned error can't find the container with id ab8dd4cd68812eface9e2ebd48b614f5a3255311e8be3ec879eff6e721653630 Nov 03 14:34:37.163001 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:37.162978 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x"] Nov 03 14:34:37.163082 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:37.163060 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:34:37.163132 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:37.163119 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:34:37.163163 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:37.163156 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:34:37.163601 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:37.163579 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:34:37.163693 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:37.163674 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:34:37.163765 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:37.163723 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:34:38.185819 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:38.185765 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" event={"ID":"50a3ab6a-5b07-4d38-b00b-9b77b506ec20","Type":"ContainerStarted","Data":"5a01c8b6ee5b43f71a8bea2356448d9f9f738e75225baffa9712cc50e7b596bd"} Nov 03 14:34:38.185819 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:38.185858 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" event={"ID":"50a3ab6a-5b07-4d38-b00b-9b77b506ec20","Type":"ContainerStarted","Data":"ab8dd4cd68812eface9e2ebd48b614f5a3255311e8be3ec879eff6e721653630"} Nov 03 14:34:38.189914 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:38.188306 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" event={"ID":"2d7d5e9c-625a-42af-98d7-52b9a8b5745e","Type":"ContainerStarted","Data":"bbc9a0ada347660eae0ce19b39a6050c28f4884c260bf3b04884871bab49c9d2"} Nov 03 14:34:38.189914 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:38.188368 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" event={"ID":"2d7d5e9c-625a-42af-98d7-52b9a8b5745e","Type":"ContainerStarted","Data":"07ee22ae1936865181729d4520c439765f167be06ce2bc9075acf98411ad6022"} Nov 03 14:34:38.292378 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:38.292326 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" podStartSLOduration=2.292310256 podStartE2EDuration="2.292310256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:34:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:34:38.267338778 +0000 UTC m=+1412.156128361" watchObservedRunningTime="2026-11-03 14:34:38.292310256 +0000 UTC m=+1412.181099831" Nov 03 14:34:38.348861 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:38.336517 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" podStartSLOduration=2.336499582 podStartE2EDuration="2.336499582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:34:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:34:38.316139957 +0000 UTC m=+1412.204929540" watchObservedRunningTime="2026-11-03 14:34:38.336499582 +0000 UTC m=+1412.225289169" Nov 03 14:34:38.772874 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:38.769966 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" Nov 03 14:34:38.772874 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:38.770100 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-rr46x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 14:34:38.772874 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:38.770147 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" podUID="50a3ab6a-5b07-4d38-b00b-9b77b506ec20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 14:34:38.780862 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:38.778901 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" Nov 03 14:34:38.780862 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:38.779023 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 14:34:38.780862 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:38.779063 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" podUID="2d7d5e9c-625a-42af-98d7-52b9a8b5745e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 14:34:39.593602 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:34:39.593552 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:34:39.784287 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:39.784221 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-rr46x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 14:34:39.805859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:39.788907 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" podUID="50a3ab6a-5b07-4d38-b00b-9b77b506ec20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 14:34:39.806414 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:39.785244 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 14:34:39.806624 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:39.806595 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" podUID="2d7d5e9c-625a-42af-98d7-52b9a8b5745e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 14:34:40.771858 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:40.771687 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-rr46x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 14:34:40.771858 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:40.771760 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" podUID="50a3ab6a-5b07-4d38-b00b-9b77b506ec20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 14:34:40.786106 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:40.786074 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 14:34:40.786364 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:40.786338 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" podUID="2d7d5e9c-625a-42af-98d7-52b9a8b5745e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 14:34:41.768442 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:41.768345 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-rr46x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 14:34:41.768442 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:41.768418 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" podUID="50a3ab6a-5b07-4d38-b00b-9b77b506ec20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 14:34:41.775731 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:41.775692 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 14:34:41.776130 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:41.775750 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" podUID="2d7d5e9c-625a-42af-98d7-52b9a8b5745e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 14:34:42.769918 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:42.768463 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-rr46x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 14:34:42.769918 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:42.768522 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" podUID="50a3ab6a-5b07-4d38-b00b-9b77b506ec20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 14:34:42.775460 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:42.775425 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 14:34:42.775578 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:42.775472 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" podUID="2d7d5e9c-625a-42af-98d7-52b9a8b5745e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 14:34:43.768688 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:43.768641 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-rr46x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 14:34:43.769328 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:43.768715 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" podUID="50a3ab6a-5b07-4d38-b00b-9b77b506ec20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 14:34:43.776059 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:43.776027 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 14:34:43.776211 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:43.776084 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" podUID="2d7d5e9c-625a-42af-98d7-52b9a8b5745e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 14:34:44.768125 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:44.768080 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-rr46x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 14:34:44.768338 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:44.768150 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" podUID="50a3ab6a-5b07-4d38-b00b-9b77b506ec20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 14:34:44.775616 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:44.775572 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 14:34:44.776143 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:44.775649 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" podUID="2d7d5e9c-625a-42af-98d7-52b9a8b5745e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 14:34:45.768756 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:45.768707 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-rr46x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 14:34:45.768974 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:45.768772 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" podUID="50a3ab6a-5b07-4d38-b00b-9b77b506ec20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 14:34:45.776147 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:45.776113 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 14:34:45.776613 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:45.776168 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" podUID="2d7d5e9c-625a-42af-98d7-52b9a8b5745e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 14:34:47.448812 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:47.448707 5060 generic.go:356] "Generic (PLEG): container finished" podID="8dc088f1-c679-4845-ba33-682ed25b6d22" containerID="86a3d9c1575232449cbfdcb3b93c0988085e6bdbc41e660126bdb4cfd6ef9af0" exitCode=0 Nov 03 14:34:47.448812 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:47.448789 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" event={"ID":"8dc088f1-c679-4845-ba33-682ed25b6d22","Type":"ContainerDied","Data":"86a3d9c1575232449cbfdcb3b93c0988085e6bdbc41e660126bdb4cfd6ef9af0"} Nov 03 14:34:47.449452 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:47.449316 5060 scope.go:122] "RemoveContainer" containerID="86a3d9c1575232449cbfdcb3b93c0988085e6bdbc41e660126bdb4cfd6ef9af0" Nov 03 14:34:47.451914 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:47.451870 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:34:47.452037 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:47.451967 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:34:47.452106 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:47.452049 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:34:47.759265 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:47.759228 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" Nov 03 14:34:47.810140 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:47.810083 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-rr46x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:34:47.810850 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:47.810801 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" podUID="50a3ab6a-5b07-4d38-b00b-9b77b506ec20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:34:47.820038 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:47.819998 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:34:47.821789 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:47.821760 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" podUID="2d7d5e9c-625a-42af-98d7-52b9a8b5745e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:34:48.206263 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:48.206177 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" Nov 03 14:34:48.207961 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:48.207941 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" Nov 03 14:34:48.208968 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:48.208947 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" Nov 03 14:34:48.296135 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:48.296100 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" Nov 03 14:34:48.296532 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:48.296510 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" Nov 03 14:34:48.297470 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:48.297446 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" Nov 03 14:34:48.457955 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:48.457866 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" event={"ID":"8dc088f1-c679-4845-ba33-682ed25b6d22","Type":"ContainerStarted","Data":"6a633b4edaf6bcd6a6979b06031c601a40071494258fc708856a6081b5be099d"} Nov 03 14:34:48.962593 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:48.962554 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" Nov 03 14:34:48.963441 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:48.963399 5060 patch_prober.go:29] interesting pod/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 14:34:48.963570 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:48.963506 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" podUID="8dc088f1-c679-4845-ba33-682ed25b6d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 14:34:49.962378 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:49.962343 5060 patch_prober.go:29] interesting pod/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 14:34:49.962822 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:49.962414 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" podUID="8dc088f1-c679-4845-ba33-682ed25b6d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 14:34:50.969856 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:50.965953 5060 patch_prober.go:29] interesting pod/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 14:34:50.969856 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:50.966013 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" podUID="8dc088f1-c679-4845-ba33-682ed25b6d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 14:34:51.968628 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:51.967410 5060 patch_prober.go:29] interesting pod/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 14:34:51.968628 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:51.967468 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" podUID="8dc088f1-c679-4845-ba33-682ed25b6d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 14:34:52.965252 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:52.965216 5060 patch_prober.go:29] interesting pod/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 14:34:52.965928 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:52.965891 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" podUID="8dc088f1-c679-4845-ba33-682ed25b6d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 14:34:53.962063 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:53.962026 5060 patch_prober.go:29] interesting pod/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 14:34:53.962345 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:53.962317 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" podUID="8dc088f1-c679-4845-ba33-682ed25b6d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 14:34:54.966693 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:54.966659 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" Nov 03 14:34:54.967120 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:54.967073 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" Nov 03 14:34:54.967932 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:34:54.967904 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" Nov 03 14:35:00.917078 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:00.917039 5060 patch_prober.go:29] interesting pod/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:35:00.917591 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:00.917117 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" podUID="7ae02904-4e94-487e-a422-2749cdb51c4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:35:03.740863 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:03.740797 5060 generic.go:356] "Generic (PLEG): container finished" podID="7ae02904-4e94-487e-a422-2749cdb51c4f" containerID="083147e066d5126568ef8097c11067f2637dd8df409af907b3d6793beb16add6" exitCode=0 Nov 03 14:35:03.741350 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:03.740871 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" event={"ID":"7ae02904-4e94-487e-a422-2749cdb51c4f","Type":"ContainerDied","Data":"083147e066d5126568ef8097c11067f2637dd8df409af907b3d6793beb16add6"} Nov 03 14:35:03.741430 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:03.741380 5060 scope.go:122] "RemoveContainer" containerID="083147e066d5126568ef8097c11067f2637dd8df409af907b3d6793beb16add6" Nov 03 14:35:03.743109 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:03.743077 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:35:03.743187 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:03.743166 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:35:03.743289 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:03.743218 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:35:04.752582 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:04.752541 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" event={"ID":"7ae02904-4e94-487e-a422-2749cdb51c4f","Type":"ContainerStarted","Data":"97f6add77a8a520b3597edfd97deb350f1c92448a8be6f96e1dae411cc8c10f4"} Nov 03 14:35:05.135080 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:05.134978 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" Nov 03 14:35:05.135424 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:05.135399 5060 patch_prober.go:29] interesting pod/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 14:35:05.135487 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:05.135450 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" podUID="7ae02904-4e94-487e-a422-2749cdb51c4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 14:35:06.131859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:06.131460 5060 patch_prober.go:29] interesting pod/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 14:35:06.131859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:06.131533 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" podUID="7ae02904-4e94-487e-a422-2749cdb51c4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 14:35:07.136859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:07.136237 5060 patch_prober.go:29] interesting pod/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 14:35:07.136859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:07.136332 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" podUID="7ae02904-4e94-487e-a422-2749cdb51c4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 14:35:07.480857 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:07.480746 5060 scope.go:122] "RemoveContainer" containerID="376627bb6ce269d70f43c84e2ae1a64ddb804da5fe8100ee4c608cd7a5c9d1a1" Nov 03 14:35:07.490966 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:07.490943 5060 scope.go:122] "RemoveContainer" containerID="77dd79ede176cfd2856a7a353b7f9972ce1cf89de321cdf80f0b5992bab0b678" Nov 03 14:35:07.516856 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:07.514284 5060 scope.go:122] "RemoveContainer" containerID="4cd0350e5d34c0c87de155c5e498a4b4b04040c21f7b3f7f3b201aa6a97c3b7a" Nov 03 14:35:07.545123 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:07.545085 5060 scope.go:122] "RemoveContainer" containerID="7e4c75cabe8c013afec09828f0f90716396847464584b526b13232e723c3714a" Nov 03 14:35:07.556676 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:07.556654 5060 scope.go:122] "RemoveContainer" containerID="9fef1ba9e2e5c153b9e65ee9fa84e1fa7b47a450fe0f7f44eafe242c41a4c6a0" Nov 03 14:35:08.132574 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:08.131981 5060 patch_prober.go:29] interesting pod/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 14:35:08.132574 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:08.132047 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" podUID="7ae02904-4e94-487e-a422-2749cdb51c4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 14:35:08.935928 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:08.935885 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl"] Nov 03 14:35:08.984107 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:08.984069 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl"] Nov 03 14:35:08.992483 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:08.984362 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl" Nov 03 14:35:09.011874 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:09.011167 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-2e8614-serving-cert\"" Nov 03 14:35:09.145462 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:09.145427 5060 patch_prober.go:29] interesting pod/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 14:35:09.145774 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:09.145726 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" podUID="7ae02904-4e94-487e-a422-2749cdb51c4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 14:35:09.171095 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:09.171044 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/232c1666-ad86-40b0-b7e5-3da4d4d00200-openshift-service-ca\") pod \"maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl\" (UID: \"232c1666-ad86-40b0-b7e5-3da4d4d00200\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl" Nov 03 14:35:09.171397 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:09.171369 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/232c1666-ad86-40b0-b7e5-3da4d4d00200-maas-api-tls\") pod \"maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl\" (UID: \"232c1666-ad86-40b0-b7e5-3da4d4d00200\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl" Nov 03 14:35:09.171676 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:09.171660 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfzhw\" (UniqueName: \"kubernetes.io/projected/232c1666-ad86-40b0-b7e5-3da4d4d00200-kube-api-access-dfzhw\") pod \"maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl\" (UID: \"232c1666-ad86-40b0-b7e5-3da4d4d00200\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl" Nov 03 14:35:09.171856 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:09.171822 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/232c1666-ad86-40b0-b7e5-3da4d4d00200-metrics-certs\") pod \"maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl\" (UID: \"232c1666-ad86-40b0-b7e5-3da4d4d00200\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl" Nov 03 14:35:09.352936 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:09.352889 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfzhw\" (UniqueName: \"kubernetes.io/projected/232c1666-ad86-40b0-b7e5-3da4d4d00200-kube-api-access-dfzhw\") pod \"maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl\" (UID: \"232c1666-ad86-40b0-b7e5-3da4d4d00200\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl" Nov 03 14:35:09.353255 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:09.353233 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/232c1666-ad86-40b0-b7e5-3da4d4d00200-metrics-certs\") pod \"maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl\" (UID: \"232c1666-ad86-40b0-b7e5-3da4d4d00200\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl" Nov 03 14:35:09.353496 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:09.353479 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/232c1666-ad86-40b0-b7e5-3da4d4d00200-openshift-service-ca\") pod \"maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl\" (UID: \"232c1666-ad86-40b0-b7e5-3da4d4d00200\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl" Nov 03 14:35:09.353626 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:09.353611 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/232c1666-ad86-40b0-b7e5-3da4d4d00200-maas-api-tls\") pod \"maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl\" (UID: \"232c1666-ad86-40b0-b7e5-3da4d4d00200\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl" Nov 03 14:35:09.353955 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:35:09.353935 5060 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-2e8614-serving-cert: secret "maas-api-e2e-api-b-2e8614-serving-cert" not found Nov 03 14:35:09.354130 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:35:09.354118 5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/232c1666-ad86-40b0-b7e5-3da4d4d00200-maas-api-tls podName:232c1666-ad86-40b0-b7e5-3da4d4d00200 nodeName:}" failed. No retries permitted until 2026-11-03 14:35:09.854094248 +0000 UTC m=+1443.742883827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/232c1666-ad86-40b0-b7e5-3da4d4d00200-maas-api-tls") pod "maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl" (UID: "232c1666-ad86-40b0-b7e5-3da4d4d00200") : secret "maas-api-e2e-api-b-2e8614-serving-cert" not found Nov 03 14:35:09.380360 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:09.355798 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/232c1666-ad86-40b0-b7e5-3da4d4d00200-openshift-service-ca\") pod \"maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl\" (UID: \"232c1666-ad86-40b0-b7e5-3da4d4d00200\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl" Nov 03 14:35:09.407724 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:09.407692 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/232c1666-ad86-40b0-b7e5-3da4d4d00200-metrics-certs\") pod \"maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl\" (UID: \"232c1666-ad86-40b0-b7e5-3da4d4d00200\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl" Nov 03 14:35:09.443665 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:09.443620 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfzhw\" (UniqueName: \"kubernetes.io/projected/232c1666-ad86-40b0-b7e5-3da4d4d00200-kube-api-access-dfzhw\") pod \"maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl\" (UID: \"232c1666-ad86-40b0-b7e5-3da4d4d00200\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl" Nov 03 14:35:09.738910 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:35:09.735312 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:35:09.886874 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:09.882973 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/232c1666-ad86-40b0-b7e5-3da4d4d00200-maas-api-tls\") pod \"maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl\" (UID: \"232c1666-ad86-40b0-b7e5-3da4d4d00200\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl" Nov 03 14:35:09.886874 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:35:09.883225 5060 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-2e8614-serving-cert: secret "maas-api-e2e-api-b-2e8614-serving-cert" not found Nov 03 14:35:09.886874 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:35:09.883291 5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/232c1666-ad86-40b0-b7e5-3da4d4d00200-maas-api-tls podName:232c1666-ad86-40b0-b7e5-3da4d4d00200 nodeName:}" failed. No retries permitted until 2026-11-03 14:35:10.883272888 +0000 UTC m=+1444.772062450 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/232c1666-ad86-40b0-b7e5-3da4d4d00200-maas-api-tls") pod "maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl" (UID: "232c1666-ad86-40b0-b7e5-3da4d4d00200") : secret "maas-api-e2e-api-b-2e8614-serving-cert" not found Nov 03 14:35:10.139150 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:10.139113 5060 patch_prober.go:29] interesting pod/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 14:35:10.139771 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:10.139734 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" podUID="7ae02904-4e94-487e-a422-2749cdb51c4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 14:35:10.203056 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:10.202956 5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-156ae1/distinct-2-w3-156ae1-kserve-5988894d76-vsdlm" podUID="59dcdef3-f88e-4463-a0dd-1917e915a516" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 14:35:10.401099 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:10.400951 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk" podUID="49b53157-30cd-4fe2-8cca-a46e7dbec6f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 14:35:10.920122 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:10.920077 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/232c1666-ad86-40b0-b7e5-3da4d4d00200-maas-api-tls\") pod \"maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl\" (UID: \"232c1666-ad86-40b0-b7e5-3da4d4d00200\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl" Nov 03 14:35:10.922651 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:10.922624 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/232c1666-ad86-40b0-b7e5-3da4d4d00200-maas-api-tls\") pod \"maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl\" (UID: \"232c1666-ad86-40b0-b7e5-3da4d4d00200\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl" Nov 03 14:35:11.136287 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:11.136249 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" Nov 03 14:35:11.136725 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:11.136692 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" Nov 03 14:35:11.137489 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:11.137468 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" Nov 03 14:35:11.214762 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:11.214662 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl" Nov 03 14:35:11.527729 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:11.527697 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl"] Nov 03 14:35:11.529737 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:35:11.529685 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232c1666_ad86_40b0_b7e5_3da4d4d00200.slice/crio-686e3e73b9a40077ced240d1cdfe26933fc32842739fa5c134d693510e88ff2a WatchSource:0}: Error finding container 686e3e73b9a40077ced240d1cdfe26933fc32842739fa5c134d693510e88ff2a: Status 404 returned error can't find the container with id 686e3e73b9a40077ced240d1cdfe26933fc32842739fa5c134d693510e88ff2a Nov 03 14:35:11.843722 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:11.843683 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl" event={"ID":"232c1666-ad86-40b0-b7e5-3da4d4d00200","Type":"ContainerStarted","Data":"686e3e73b9a40077ced240d1cdfe26933fc32842739fa5c134d693510e88ff2a"} Nov 03 14:35:12.853212 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:12.853173 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl" event={"ID":"232c1666-ad86-40b0-b7e5-3da4d4d00200","Type":"ContainerStarted","Data":"b4d3564916fb4a0dfd65688d8c1fe352f0a896af24ae09b4648f448e3183d79a"} Nov 03 14:35:12.853662 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:12.853255 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl" Nov 03 14:35:12.909016 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:12.908944 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl" podStartSLOduration=4.62435395 podStartE2EDuration="4.90892281s" totalImagesPullingTime="284.56886ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:35:12.903100458 +0000 UTC m=+1446.791890039" watchObservedRunningTime="2026-11-03 14:35:12.90892281 +0000 UTC m=+1446.797712391" Nov 03 14:35:13.637895 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:13.637804 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-1ffaa0/sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk"] Nov 03 14:35:18.877132 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:18.877098 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl" Nov 03 14:35:23.806976 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:23.806931 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-2d0da2/sub-test-model-2d0da2-kserve-6f7f89db79-kffdh"] Nov 03 14:35:28.683554 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:28.683500 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk" podUID="49b53157-30cd-4fe2-8cca-a46e7dbec6f3" containerName="main" containerID="cri-o://9d99b2f5d9ba9310223e64032bce1614ee059976537d1fee7924d31bff767b4b" gracePeriod=45 Nov 03 14:35:29.063859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:29.063491 5060 generic.go:356] "Generic (PLEG): container finished" podID="49b53157-30cd-4fe2-8cca-a46e7dbec6f3" containerID="9d99b2f5d9ba9310223e64032bce1614ee059976537d1fee7924d31bff767b4b" exitCode=0 Nov 03 14:35:29.063859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:29.063683 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk" event={"ID":"49b53157-30cd-4fe2-8cca-a46e7dbec6f3","Type":"ContainerDied","Data":"9d99b2f5d9ba9310223e64032bce1614ee059976537d1fee7924d31bff767b4b"} Nov 03 14:35:29.586403 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:29.586251 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk" Nov 03 14:35:29.688909 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:29.688704 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49b53157-30cd-4fe2-8cca-a46e7dbec6f3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/49b53157-30cd-4fe2-8cca-a46e7dbec6f3-dshm\") pod \"49b53157-30cd-4fe2-8cca-a46e7dbec6f3\" (UID: \"49b53157-30cd-4fe2-8cca-a46e7dbec6f3\") " Nov 03 14:35:29.688909 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:29.688793 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49b53157-30cd-4fe2-8cca-a46e7dbec6f3-kube-api-access-k2pzr\" (UniqueName: \"kubernetes.io/projected/49b53157-30cd-4fe2-8cca-a46e7dbec6f3-kube-api-access-k2pzr\") pod \"49b53157-30cd-4fe2-8cca-a46e7dbec6f3\" (UID: \"49b53157-30cd-4fe2-8cca-a46e7dbec6f3\") " Nov 03 14:35:29.688909 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:29.688904 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49b53157-30cd-4fe2-8cca-a46e7dbec6f3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49b53157-30cd-4fe2-8cca-a46e7dbec6f3-tmp-dir\") pod \"49b53157-30cd-4fe2-8cca-a46e7dbec6f3\" (UID: \"49b53157-30cd-4fe2-8cca-a46e7dbec6f3\") " Nov 03 14:35:29.689529 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:29.689002 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49b53157-30cd-4fe2-8cca-a46e7dbec6f3-tls-certs\" (UniqueName: \"kubernetes.io/secret/49b53157-30cd-4fe2-8cca-a46e7dbec6f3-tls-certs\") pod \"49b53157-30cd-4fe2-8cca-a46e7dbec6f3\" (UID: \"49b53157-30cd-4fe2-8cca-a46e7dbec6f3\") " Nov 03 14:35:29.689529 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:29.689075 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49b53157-30cd-4fe2-8cca-a46e7dbec6f3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49b53157-30cd-4fe2-8cca-a46e7dbec6f3-model-cache\") pod \"49b53157-30cd-4fe2-8cca-a46e7dbec6f3\" (UID: \"49b53157-30cd-4fe2-8cca-a46e7dbec6f3\") " Nov 03 14:35:29.689529 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:29.689106 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49b53157-30cd-4fe2-8cca-a46e7dbec6f3-home\" (UniqueName: \"kubernetes.io/empty-dir/49b53157-30cd-4fe2-8cca-a46e7dbec6f3-home\") pod \"49b53157-30cd-4fe2-8cca-a46e7dbec6f3\" (UID: \"49b53157-30cd-4fe2-8cca-a46e7dbec6f3\") " Nov 03 14:35:29.690052 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:29.690023 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49b53157-30cd-4fe2-8cca-a46e7dbec6f3-home" pod "49b53157-30cd-4fe2-8cca-a46e7dbec6f3" (UID: "49b53157-30cd-4fe2-8cca-a46e7dbec6f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:35:29.690456 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:29.690430 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49b53157-30cd-4fe2-8cca-a46e7dbec6f3-tmp-dir" pod "49b53157-30cd-4fe2-8cca-a46e7dbec6f3" (UID: "49b53157-30cd-4fe2-8cca-a46e7dbec6f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:35:29.693093 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:29.693063 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49b53157-30cd-4fe2-8cca-a46e7dbec6f3-model-cache" pod "49b53157-30cd-4fe2-8cca-a46e7dbec6f3" (UID: "49b53157-30cd-4fe2-8cca-a46e7dbec6f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:35:29.764863 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:29.759145 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49b53157-30cd-4fe2-8cca-a46e7dbec6f3-tls-certs" pod "49b53157-30cd-4fe2-8cca-a46e7dbec6f3" (UID: "49b53157-30cd-4fe2-8cca-a46e7dbec6f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:35:29.781872 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:29.780969 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49b53157-30cd-4fe2-8cca-a46e7dbec6f3-dshm" pod "49b53157-30cd-4fe2-8cca-a46e7dbec6f3" (UID: "49b53157-30cd-4fe2-8cca-a46e7dbec6f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:35:29.781872 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:29.781388 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49b53157-30cd-4fe2-8cca-a46e7dbec6f3-kube-api-access-k2pzr" pod "49b53157-30cd-4fe2-8cca-a46e7dbec6f3" (UID: "49b53157-30cd-4fe2-8cca-a46e7dbec6f3"). InnerVolumeSpecName "kube-api-access-k2pzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:35:29.794356 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:29.791390 5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49b53157-30cd-4fe2-8cca-a46e7dbec6f3-tmp-dir\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:35:29.794356 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:29.791429 5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49b53157-30cd-4fe2-8cca-a46e7dbec6f3-tls-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:35:29.794356 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:29.791444 5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49b53157-30cd-4fe2-8cca-a46e7dbec6f3-model-cache\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:35:29.794356 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:29.791456 5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49b53157-30cd-4fe2-8cca-a46e7dbec6f3-home\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:35:29.794356 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:29.791468 5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49b53157-30cd-4fe2-8cca-a46e7dbec6f3-dshm\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:35:29.794356 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:29.791483 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2pzr\" (UniqueName: \"kubernetes.io/projected/49b53157-30cd-4fe2-8cca-a46e7dbec6f3-kube-api-access-k2pzr\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:35:30.107860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:30.106905 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk" event={"ID":"49b53157-30cd-4fe2-8cca-a46e7dbec6f3","Type":"ContainerDied","Data":"30d6b36bef969ab95570d78e5061e37d589bfaa1090aac8f6681db7a8bd6476f"} Nov 03 14:35:30.107860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:30.106989 5060 scope.go:122] "RemoveContainer" containerID="9d99b2f5d9ba9310223e64032bce1614ee059976537d1fee7924d31bff767b4b" Nov 03 14:35:30.107860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:30.107243 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk" Nov 03 14:35:30.294063 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:30.294025 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-1ffaa0/sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk"] Nov 03 14:35:30.294063 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:30.294062 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-1ffaa0/sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk"] Nov 03 14:35:30.394171 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:30.394047 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/sub-test-model-1ffaa0-kserve-5f549b4f5-jjwvk" podUID="49b53157-30cd-4fe2-8cca-a46e7dbec6f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 14:35:32.250666 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:32.250626 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49b53157-30cd-4fe2-8cca-a46e7dbec6f3" path="/var/lib/kubelet/pods/49b53157-30cd-4fe2-8cca-a46e7dbec6f3/volumes" Nov 03 14:35:32.318554 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:32.318517 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28"] Nov 03 14:35:32.319166 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:32.319150 5060 cpu_manager.go:414] "containerMap: removing container" podUID="49b53157-30cd-4fe2-8cca-a46e7dbec6f3" containerName="main" Nov 03 14:35:32.319225 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:32.319168 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49b53157-30cd-4fe2-8cca-a46e7dbec6f3" containerName="main" Nov 03 14:35:32.319297 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:32.319288 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="49b53157-30cd-4fe2-8cca-a46e7dbec6f3" containerName="main" Nov 03 14:35:32.324324 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:32.324304 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" Nov 03 14:35:32.350116 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:32.350084 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28"] Nov 03 14:35:32.442964 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:32.442929 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-workload-socket\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28\" (UID: \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" Nov 03 14:35:32.443163 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:32.442985 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-credential-socket\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28\" (UID: \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" Nov 03 14:35:32.443163 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:32.443002 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-workload-certs\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28\" (UID: \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" Nov 03 14:35:32.443163 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:32.443143 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-istiod-ca-cert\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28\" (UID: \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" Nov 03 14:35:32.443289 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:32.443197 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjd5p\" (UniqueName: \"kubernetes.io/projected/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-kube-api-access-wjd5p\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28\" (UID: \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" Nov 03 14:35:32.443289 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:32.443250 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-istio-token\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28\" (UID: \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" Nov 03 14:35:32.443371 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:32.443332 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-istio-envoy\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28\" (UID: \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" Nov 03 14:35:32.443408 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:32.443373 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-istio-podinfo\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28\" (UID: \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" Nov 03 14:35:32.443454 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:32.443420 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-istio-data\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28\" (UID: \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" Nov 03 14:35:32.544966 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:32.544874 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-istiod-ca-cert\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28\" (UID: \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" Nov 03 14:35:32.544966 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:32.544949 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjd5p\" (UniqueName: \"kubernetes.io/projected/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-kube-api-access-wjd5p\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28\" (UID: \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" Nov 03 14:35:32.545228 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:32.544995 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-istio-token\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28\" (UID: \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" Nov 03 14:35:32.545228 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:32.545099 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-istio-envoy\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28\" (UID: \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" Nov 03 14:35:32.545228 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:32.545153 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-istio-podinfo\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28\" (UID: \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" Nov 03 14:35:32.545388 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:32.545236 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-istio-data\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28\" (UID: \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" Nov 03 14:35:32.545388 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:32.545296 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-workload-socket\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28\" (UID: \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" Nov 03 14:35:32.545388 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:32.545359 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-credential-socket\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28\" (UID: \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" Nov 03 14:35:32.545388 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:32.545384 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-workload-certs\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28\" (UID: \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" Nov 03 14:35:32.545646 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:32.545622 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-istio-data\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28\" (UID: \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" Nov 03 14:35:32.545739 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:32.545709 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-credential-socket\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28\" (UID: \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" Nov 03 14:35:32.545739 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:32.545734 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-workload-socket\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28\" (UID: \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" Nov 03 14:35:32.545911 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:32.545734 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-istiod-ca-cert\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28\" (UID: \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" Nov 03 14:35:32.545911 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:32.545823 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-workload-certs\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28\" (UID: \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" Nov 03 14:35:32.547379 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:32.547359 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-istio-envoy\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28\" (UID: \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" Nov 03 14:35:32.547689 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:32.547669 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-istio-podinfo\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28\" (UID: \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" Nov 03 14:35:32.568440 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:32.568401 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjd5p\" (UniqueName: \"kubernetes.io/projected/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-kube-api-access-wjd5p\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28\" (UID: \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" Nov 03 14:35:32.568569 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:32.568517 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-istio-token\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28\" (UID: \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" Nov 03 14:35:32.642876 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:32.642829 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" Nov 03 14:35:33.384511 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:33.384466 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28"] Nov 03 14:35:33.389017 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:33.388953 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:35:33.389223 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:33.389199 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:35:33.389367 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:33.389346 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:35:34.172859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:34.172025 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" event={"ID":"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7","Type":"ContainerStarted","Data":"e0aae71cc1e2efea9cb4773f11bae8882d915dd610406a7bf19bde889f1d44d2"} Nov 03 14:35:34.172859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:34.172124 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" event={"ID":"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7","Type":"ContainerStarted","Data":"519ff47f1e4a0878d3cf111b9bdc8fe63e8bea5dfe9853479684071c14987eba"} Nov 03 14:35:34.236921 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:34.235507 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" podStartSLOduration=2.235488287 podStartE2EDuration="2.235488287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:35:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:35:34.23200998 +0000 UTC m=+1468.120799562" watchObservedRunningTime="2026-11-03 14:35:34.235488287 +0000 UTC m=+1468.124277870" Nov 03 14:35:34.647626 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:34.647589 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" Nov 03 14:35:34.648498 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:34.648471 5060 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 14:35:34.648677 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:34.648653 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" podUID="2aafae9b-8aa6-46fe-bb81-a37c0e52fff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 14:35:35.665859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:35.664986 5060 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 14:35:35.665859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:35.665057 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" podUID="2aafae9b-8aa6-46fe-bb81-a37c0e52fff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 14:35:36.643317 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:36.643275 5060 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 14:35:36.643604 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:36.643571 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" podUID="2aafae9b-8aa6-46fe-bb81-a37c0e52fff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 14:35:37.648860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:37.644862 5060 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 14:35:37.648860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:37.644931 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" podUID="2aafae9b-8aa6-46fe-bb81-a37c0e52fff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 14:35:38.658858 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:38.655277 5060 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 14:35:38.658858 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:38.655359 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" podUID="2aafae9b-8aa6-46fe-bb81-a37c0e52fff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 14:35:39.020420 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:39.016247 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-2d0da2/sub-test-model-2d0da2-kserve-6f7f89db79-kffdh" podUID="3dc0b682-756a-4736-9d3a-ab20195b4bcb" containerName="main" containerID="cri-o://d0076503938c1f199853bd33c586b9bce5b58721880edcc552d3062bd2f3c706" gracePeriod=45 Nov 03 14:35:39.526859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:39.520221 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98" podUID="3aaaeea4-134f-4d17-a180-b4857c6425d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 14:35:39.615860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:39.607436 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-2d0da2/sub-test-model-2d0da2-kserve-6f7f89db79-kffdh" podUID="3dc0b682-756a-4736-9d3a-ab20195b4bcb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 14:35:39.615860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:39.612523 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-2d0da2/sub-test-model-2d0da2-kserve-6f7f89db79-kffdh" podUID="3dc0b682-756a-4736-9d3a-ab20195b4bcb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": dial tcp 10.132.0.103:8000: connect: connection refused" Nov 03 14:35:39.615860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:39.612668 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-2d0da2/sub-test-model-2d0da2-kserve-6f7f89db79-kffdh" Nov 03 14:35:39.644285 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:39.644241 5060 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 14:35:39.644538 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:39.644511 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" podUID="2aafae9b-8aa6-46fe-bb81-a37c0e52fff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 14:35:39.696921 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:35:39.696889 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:35:40.433502 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:40.433473 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-2d0da2/sub-test-model-2d0da2-kserve-6f7f89db79-kffdh" Nov 03 14:35:40.532710 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:40.531440 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dc0b682-756a-4736-9d3a-ab20195b4bcb-tls-certs\" (UniqueName: \"kubernetes.io/secret/3dc0b682-756a-4736-9d3a-ab20195b4bcb-tls-certs\") pod \"3dc0b682-756a-4736-9d3a-ab20195b4bcb\" (UID: \"3dc0b682-756a-4736-9d3a-ab20195b4bcb\") " Nov 03 14:35:40.532710 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:40.531570 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dc0b682-756a-4736-9d3a-ab20195b4bcb-home\" (UniqueName: \"kubernetes.io/empty-dir/3dc0b682-756a-4736-9d3a-ab20195b4bcb-home\") pod \"3dc0b682-756a-4736-9d3a-ab20195b4bcb\" (UID: \"3dc0b682-756a-4736-9d3a-ab20195b4bcb\") " Nov 03 14:35:40.532710 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:40.531616 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dc0b682-756a-4736-9d3a-ab20195b4bcb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dc0b682-756a-4736-9d3a-ab20195b4bcb-dshm\") pod \"3dc0b682-756a-4736-9d3a-ab20195b4bcb\" (UID: \"3dc0b682-756a-4736-9d3a-ab20195b4bcb\") " Nov 03 14:35:40.532710 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:40.531672 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dc0b682-756a-4736-9d3a-ab20195b4bcb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dc0b682-756a-4736-9d3a-ab20195b4bcb-model-cache\") pod \"3dc0b682-756a-4736-9d3a-ab20195b4bcb\" (UID: \"3dc0b682-756a-4736-9d3a-ab20195b4bcb\") " Nov 03 14:35:40.532710 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:40.531734 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dc0b682-756a-4736-9d3a-ab20195b4bcb-kube-api-access-m5xtp\" (UniqueName: \"kubernetes.io/projected/3dc0b682-756a-4736-9d3a-ab20195b4bcb-kube-api-access-m5xtp\") pod \"3dc0b682-756a-4736-9d3a-ab20195b4bcb\" (UID: \"3dc0b682-756a-4736-9d3a-ab20195b4bcb\") " Nov 03 14:35:40.532710 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:40.531969 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dc0b682-756a-4736-9d3a-ab20195b4bcb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dc0b682-756a-4736-9d3a-ab20195b4bcb-tmp-dir\") pod \"3dc0b682-756a-4736-9d3a-ab20195b4bcb\" (UID: \"3dc0b682-756a-4736-9d3a-ab20195b4bcb\") " Nov 03 14:35:40.533345 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:40.532940 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dc0b682-756a-4736-9d3a-ab20195b4bcb-home" pod "3dc0b682-756a-4736-9d3a-ab20195b4bcb" (UID: "3dc0b682-756a-4736-9d3a-ab20195b4bcb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:35:40.533345 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:40.533298 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dc0b682-756a-4736-9d3a-ab20195b4bcb-model-cache" pod "3dc0b682-756a-4736-9d3a-ab20195b4bcb" (UID: "3dc0b682-756a-4736-9d3a-ab20195b4bcb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:35:40.533546 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:40.533507 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dc0b682-756a-4736-9d3a-ab20195b4bcb-tmp-dir" pod "3dc0b682-756a-4736-9d3a-ab20195b4bcb" (UID: "3dc0b682-756a-4736-9d3a-ab20195b4bcb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:35:40.535189 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:40.535165 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dc0b682-756a-4736-9d3a-ab20195b4bcb-dshm" pod "3dc0b682-756a-4736-9d3a-ab20195b4bcb" (UID: "3dc0b682-756a-4736-9d3a-ab20195b4bcb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:35:40.535971 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:40.535936 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dc0b682-756a-4736-9d3a-ab20195b4bcb-kube-api-access-m5xtp" pod "3dc0b682-756a-4736-9d3a-ab20195b4bcb" (UID: "3dc0b682-756a-4736-9d3a-ab20195b4bcb"). InnerVolumeSpecName "kube-api-access-m5xtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:35:40.536065 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:40.535968 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dc0b682-756a-4736-9d3a-ab20195b4bcb-tls-certs" pod "3dc0b682-756a-4736-9d3a-ab20195b4bcb" (UID: "3dc0b682-756a-4736-9d3a-ab20195b4bcb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:35:40.561370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:40.557495 5060 generic.go:356] "Generic (PLEG): container finished" podID="3dc0b682-756a-4736-9d3a-ab20195b4bcb" containerID="d0076503938c1f199853bd33c586b9bce5b58721880edcc552d3062bd2f3c706" exitCode=0 Nov 03 14:35:40.561370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:40.557654 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-2d0da2/sub-test-model-2d0da2-kserve-6f7f89db79-kffdh" event={"ID":"3dc0b682-756a-4736-9d3a-ab20195b4bcb","Type":"ContainerDied","Data":"d0076503938c1f199853bd33c586b9bce5b58721880edcc552d3062bd2f3c706"} Nov 03 14:35:40.561370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:40.557692 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-2d0da2/sub-test-model-2d0da2-kserve-6f7f89db79-kffdh" event={"ID":"3dc0b682-756a-4736-9d3a-ab20195b4bcb","Type":"ContainerDied","Data":"e2a8b5592ab8c638035c1f317018983df28b1371365cd55bbebdfa7eb7a624b4"} Nov 03 14:35:40.561370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:40.557716 5060 scope.go:122] "RemoveContainer" containerID="d0076503938c1f199853bd33c586b9bce5b58721880edcc552d3062bd2f3c706" Nov 03 14:35:40.561370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:40.557938 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-2d0da2/sub-test-model-2d0da2-kserve-6f7f89db79-kffdh" Nov 03 14:35:40.573371 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:40.573350 5060 scope.go:122] "RemoveContainer" containerID="d0076503938c1f199853bd33c586b9bce5b58721880edcc552d3062bd2f3c706" Nov 03 14:35:40.573764 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:35:40.573730 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0076503938c1f199853bd33c586b9bce5b58721880edcc552d3062bd2f3c706\": container with ID starting with d0076503938c1f199853bd33c586b9bce5b58721880edcc552d3062bd2f3c706 not found: ID does not exist" containerID="d0076503938c1f199853bd33c586b9bce5b58721880edcc552d3062bd2f3c706" Nov 03 14:35:40.573896 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:40.573775 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0076503938c1f199853bd33c586b9bce5b58721880edcc552d3062bd2f3c706"} err="failed to get container status \"d0076503938c1f199853bd33c586b9bce5b58721880edcc552d3062bd2f3c706\": rpc error: code = NotFound desc = could not find container \"d0076503938c1f199853bd33c586b9bce5b58721880edcc552d3062bd2f3c706\": container with ID starting with d0076503938c1f199853bd33c586b9bce5b58721880edcc552d3062bd2f3c706 not found: ID does not exist" Nov 03 14:35:40.633596 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:40.633557 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-2d0da2/sub-test-model-2d0da2-kserve-6f7f89db79-kffdh"] Nov 03 14:35:40.634149 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:40.634120 5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dc0b682-756a-4736-9d3a-ab20195b4bcb-tmp-dir\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:35:40.634149 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:40.634151 5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dc0b682-756a-4736-9d3a-ab20195b4bcb-tls-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:35:40.634304 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:40.634164 5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dc0b682-756a-4736-9d3a-ab20195b4bcb-home\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:35:40.634304 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:40.634174 5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dc0b682-756a-4736-9d3a-ab20195b4bcb-dshm\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:35:40.634304 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:40.634185 5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dc0b682-756a-4736-9d3a-ab20195b4bcb-model-cache\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:35:40.634304 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:40.634196 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5xtp\" (UniqueName: \"kubernetes.io/projected/3dc0b682-756a-4736-9d3a-ab20195b4bcb-kube-api-access-m5xtp\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:35:40.636864 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:40.636819 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-2d0da2/sub-test-model-2d0da2-kserve-6f7f89db79-kffdh"] Nov 03 14:35:40.643924 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:40.643903 5060 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 14:35:40.644004 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:40.643945 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" podUID="2aafae9b-8aa6-46fe-bb81-a37c0e52fff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 14:35:41.648208 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:41.648179 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" Nov 03 14:35:41.648698 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:41.648678 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" Nov 03 14:35:41.649369 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:41.649344 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" Nov 03 14:35:42.249617 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:42.249584 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dc0b682-756a-4736-9d3a-ab20195b4bcb" path="/var/lib/kubelet/pods/3dc0b682-756a-4736-9d3a-ab20195b4bcb/volumes" Nov 03 14:35:49.269475 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:49.268938 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-rr46x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:35:49.269475 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:49.269006 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" podUID="50a3ab6a-5b07-4d38-b00b-9b77b506ec20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:35:49.344315 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:49.343970 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:35:49.344315 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:49.344036 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" podUID="2d7d5e9c-625a-42af-98d7-52b9a8b5745e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:35:52.040027 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:52.039989 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a35a5574-sxdjh"] Nov 03 14:35:52.052690 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:52.052658 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a35a5574-sxdjh"] Nov 03 14:35:52.268734 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:35:52.268702 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dea710fa-d5e0-463e-adff-8d708346f953" path="/var/lib/kubelet/pods/dea710fa-d5e0-463e-adff-8d708346f953/volumes" Nov 03 14:36:04.853859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:04.849982 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6"] Nov 03 14:36:04.861855 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:04.858130 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6" podUID="b93ca36c-db33-4a19-b384-331987d1bb9f" containerName="maas-api" containerID="cri-o://b446a8eec756fc59ff8636739e52bf8f77a36d0f4a5d20c9c08e59316cea5ad5" gracePeriod=30 Nov 03 14:36:05.068588 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:05.068547 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-054f11b4-r459z"] Nov 03 14:36:05.069745 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:05.069713 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-054f11b4-r459z"] Nov 03 14:36:05.371635 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:05.324984 5060 generic.go:356] "Generic (PLEG): container finished" podID="b93ca36c-db33-4a19-b384-331987d1bb9f" containerID="b446a8eec756fc59ff8636739e52bf8f77a36d0f4a5d20c9c08e59316cea5ad5" exitCode=0 Nov 03 14:36:05.371635 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:05.325120 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6" event={"ID":"b93ca36c-db33-4a19-b384-331987d1bb9f","Type":"ContainerDied","Data":"b446a8eec756fc59ff8636739e52bf8f77a36d0f4a5d20c9c08e59316cea5ad5"} Nov 03 14:36:06.614936 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:06.602265 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6" Nov 03 14:36:06.707942 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:06.703734 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b93ca36c-db33-4a19-b384-331987d1bb9f-kube-api-access-zvvpn\" (UniqueName: \"kubernetes.io/projected/b93ca36c-db33-4a19-b384-331987d1bb9f-kube-api-access-zvvpn\") pod \"b93ca36c-db33-4a19-b384-331987d1bb9f\" (UID: \"b93ca36c-db33-4a19-b384-331987d1bb9f\") " Nov 03 14:36:06.707942 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:06.703800 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b93ca36c-db33-4a19-b384-331987d1bb9f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b93ca36c-db33-4a19-b384-331987d1bb9f-maas-api-tls\") pod \"b93ca36c-db33-4a19-b384-331987d1bb9f\" (UID: \"b93ca36c-db33-4a19-b384-331987d1bb9f\") " Nov 03 14:36:06.707942 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:06.703907 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b93ca36c-db33-4a19-b384-331987d1bb9f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b93ca36c-db33-4a19-b384-331987d1bb9f-openshift-service-ca\") pod \"b93ca36c-db33-4a19-b384-331987d1bb9f\" (UID: \"b93ca36c-db33-4a19-b384-331987d1bb9f\") " Nov 03 14:36:06.707942 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:06.704098 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b93ca36c-db33-4a19-b384-331987d1bb9f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b93ca36c-db33-4a19-b384-331987d1bb9f-metrics-certs\") pod \"b93ca36c-db33-4a19-b384-331987d1bb9f\" (UID: \"b93ca36c-db33-4a19-b384-331987d1bb9f\") " Nov 03 14:36:06.727291 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:06.727237 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b93ca36c-db33-4a19-b384-331987d1bb9f-openshift-service-ca" pod "b93ca36c-db33-4a19-b384-331987d1bb9f" (UID: "b93ca36c-db33-4a19-b384-331987d1bb9f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:36:06.750821 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:06.744951 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b93ca36c-db33-4a19-b384-331987d1bb9f-metrics-certs" pod "b93ca36c-db33-4a19-b384-331987d1bb9f" (UID: "b93ca36c-db33-4a19-b384-331987d1bb9f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:36:06.758684 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:06.758623 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b93ca36c-db33-4a19-b384-331987d1bb9f-maas-api-tls" pod "b93ca36c-db33-4a19-b384-331987d1bb9f" (UID: "b93ca36c-db33-4a19-b384-331987d1bb9f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:36:06.770084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:06.770049 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b93ca36c-db33-4a19-b384-331987d1bb9f-kube-api-access-zvvpn" pod "b93ca36c-db33-4a19-b384-331987d1bb9f" (UID: "b93ca36c-db33-4a19-b384-331987d1bb9f"). InnerVolumeSpecName "kube-api-access-zvvpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:36:06.776410 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:06.776369 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f08e371c-e230-4bcf-9e3d-114e6c81c7d2" path="/var/lib/kubelet/pods/f08e371c-e230-4bcf-9e3d-114e6c81c7d2/volumes" Nov 03 14:36:06.777485 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:06.777447 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6" event={"ID":"b93ca36c-db33-4a19-b384-331987d1bb9f","Type":"ContainerDied","Data":"c3b40240082fb10e0562f2b8b4e4bb3270847431c5bbe3f1970a5245845761e9"} Nov 03 14:36:06.777706 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:06.777677 5060 scope.go:122] "RemoveContainer" containerID="b446a8eec756fc59ff8636739e52bf8f77a36d0f4a5d20c9c08e59316cea5ad5" Nov 03 14:36:06.843078 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:06.816732 5060 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b93ca36c-db33-4a19-b384-331987d1bb9f-openshift-service-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:36:06.843078 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:06.816763 5060 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b93ca36c-db33-4a19-b384-331987d1bb9f-metrics-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:36:06.843078 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:06.816779 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvvpn\" (UniqueName: \"kubernetes.io/projected/b93ca36c-db33-4a19-b384-331987d1bb9f-kube-api-access-zvvpn\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:36:06.843078 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:06.816791 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b93ca36c-db33-4a19-b384-331987d1bb9f-maas-api-tls\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:36:07.375027 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:07.374982 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/3.log" Nov 03 14:36:07.385530 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:07.385506 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/3.log" Nov 03 14:36:07.617388 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:07.617343 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6" Nov 03 14:36:07.654818 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:07.654787 5060 scope.go:122] "RemoveContainer" containerID="aa36f7a6421720e925dfbdc3c96f13a3f29dea58ce5ff1f53762cea4e80d2285" Nov 03 14:36:07.666697 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:07.666674 5060 scope.go:122] "RemoveContainer" containerID="2d3f860a4d1e564b8e3dbb06f6a357a2f174ef50012cbd65dbe3cab55f750bf1" Nov 03 14:36:07.678621 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:07.678581 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6"] Nov 03 14:36:07.690597 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:07.690569 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cb587c-864c875d7f-bfrl6"] Nov 03 14:36:08.250190 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:08.250157 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b93ca36c-db33-4a19-b384-331987d1bb9f" path="/var/lib/kubelet/pods/b93ca36c-db33-4a19-b384-331987d1bb9f/volumes" Nov 03 14:36:09.440491 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:36:09.440456 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:36:15.427521 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:15.423595 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc"] Nov 03 14:36:15.427521 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:15.424373 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" podUID="8dc088f1-c679-4845-ba33-682ed25b6d22" containerName="istio-proxy" containerID="cri-o://6a633b4edaf6bcd6a6979b06031c601a40071494258fc708856a6081b5be099d" gracePeriod=30 Nov 03 14:36:20.714160 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:20.714092 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" Nov 03 14:36:20.756846 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:20.756802 5060 generic.go:356] "Generic (PLEG): container finished" podID="8dc088f1-c679-4845-ba33-682ed25b6d22" containerID="6a633b4edaf6bcd6a6979b06031c601a40071494258fc708856a6081b5be099d" exitCode=0 Nov 03 14:36:20.756995 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:20.756883 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" event={"ID":"8dc088f1-c679-4845-ba33-682ed25b6d22","Type":"ContainerDied","Data":"6a633b4edaf6bcd6a6979b06031c601a40071494258fc708856a6081b5be099d"} Nov 03 14:36:20.756995 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:20.756890 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" Nov 03 14:36:20.756995 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:20.756955 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc" event={"ID":"8dc088f1-c679-4845-ba33-682ed25b6d22","Type":"ContainerDied","Data":"359cb1ec5e61383b6987fb927b19b997f58f97777673212e2618626924081e05"} Nov 03 14:36:20.756995 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:20.756988 5060 scope.go:122] "RemoveContainer" containerID="6a633b4edaf6bcd6a6979b06031c601a40071494258fc708856a6081b5be099d" Nov 03 14:36:20.767008 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:20.766990 5060 scope.go:122] "RemoveContainer" containerID="86a3d9c1575232449cbfdcb3b93c0988085e6bdbc41e660126bdb4cfd6ef9af0" Nov 03 14:36:20.775744 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:20.775726 5060 scope.go:122] "RemoveContainer" containerID="6a633b4edaf6bcd6a6979b06031c601a40071494258fc708856a6081b5be099d" Nov 03 14:36:20.776027 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:36:20.776009 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a633b4edaf6bcd6a6979b06031c601a40071494258fc708856a6081b5be099d\": container with ID starting with 6a633b4edaf6bcd6a6979b06031c601a40071494258fc708856a6081b5be099d not found: ID does not exist" containerID="6a633b4edaf6bcd6a6979b06031c601a40071494258fc708856a6081b5be099d" Nov 03 14:36:20.776091 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:20.776038 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a633b4edaf6bcd6a6979b06031c601a40071494258fc708856a6081b5be099d"} err="failed to get container status \"6a633b4edaf6bcd6a6979b06031c601a40071494258fc708856a6081b5be099d\": rpc error: code = NotFound desc = could not find container \"6a633b4edaf6bcd6a6979b06031c601a40071494258fc708856a6081b5be099d\": container with ID starting with 6a633b4edaf6bcd6a6979b06031c601a40071494258fc708856a6081b5be099d not found: ID does not exist" Nov 03 14:36:20.776091 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:20.776055 5060 scope.go:122] "RemoveContainer" containerID="86a3d9c1575232449cbfdcb3b93c0988085e6bdbc41e660126bdb4cfd6ef9af0" Nov 03 14:36:20.776288 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:36:20.776271 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86a3d9c1575232449cbfdcb3b93c0988085e6bdbc41e660126bdb4cfd6ef9af0\": container with ID starting with 86a3d9c1575232449cbfdcb3b93c0988085e6bdbc41e660126bdb4cfd6ef9af0 not found: ID does not exist" containerID="86a3d9c1575232449cbfdcb3b93c0988085e6bdbc41e660126bdb4cfd6ef9af0" Nov 03 14:36:20.776330 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:20.776294 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86a3d9c1575232449cbfdcb3b93c0988085e6bdbc41e660126bdb4cfd6ef9af0"} err="failed to get container status \"86a3d9c1575232449cbfdcb3b93c0988085e6bdbc41e660126bdb4cfd6ef9af0\": rpc error: code = NotFound desc = could not find container \"86a3d9c1575232449cbfdcb3b93c0988085e6bdbc41e660126bdb4cfd6ef9af0\": container with ID starting with 86a3d9c1575232449cbfdcb3b93c0988085e6bdbc41e660126bdb4cfd6ef9af0 not found: ID does not exist" Nov 03 14:36:20.846383 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:20.846351 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8dc088f1-c679-4845-ba33-682ed25b6d22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dc088f1-c679-4845-ba33-682ed25b6d22-istio-podinfo\") pod \"8dc088f1-c679-4845-ba33-682ed25b6d22\" (UID: \"8dc088f1-c679-4845-ba33-682ed25b6d22\") " Nov 03 14:36:20.846588 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:20.846404 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dc088f1-c679-4845-ba33-682ed25b6d22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dc088f1-c679-4845-ba33-682ed25b6d22-istio-data\") pod \"8dc088f1-c679-4845-ba33-682ed25b6d22\" (UID: \"8dc088f1-c679-4845-ba33-682ed25b6d22\") " Nov 03 14:36:20.846588 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:20.846453 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dc088f1-c679-4845-ba33-682ed25b6d22-kube-api-access-8gdsq\" (UniqueName: \"kubernetes.io/projected/8dc088f1-c679-4845-ba33-682ed25b6d22-kube-api-access-8gdsq\") pod \"8dc088f1-c679-4845-ba33-682ed25b6d22\" (UID: \"8dc088f1-c679-4845-ba33-682ed25b6d22\") " Nov 03 14:36:20.846588 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:20.846573 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dc088f1-c679-4845-ba33-682ed25b6d22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dc088f1-c679-4845-ba33-682ed25b6d22-istio-envoy\") pod \"8dc088f1-c679-4845-ba33-682ed25b6d22\" (UID: \"8dc088f1-c679-4845-ba33-682ed25b6d22\") " Nov 03 14:36:20.846791 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:20.846761 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dc088f1-c679-4845-ba33-682ed25b6d22-istio-data" pod "8dc088f1-c679-4845-ba33-682ed25b6d22" (UID: "8dc088f1-c679-4845-ba33-682ed25b6d22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:36:20.847004 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:20.846982 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dc088f1-c679-4845-ba33-682ed25b6d22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dc088f1-c679-4845-ba33-682ed25b6d22-credential-socket\") pod \"8dc088f1-c679-4845-ba33-682ed25b6d22\" (UID: \"8dc088f1-c679-4845-ba33-682ed25b6d22\") " Nov 03 14:36:20.847334 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:20.847238 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dc088f1-c679-4845-ba33-682ed25b6d22-credential-socket" pod "8dc088f1-c679-4845-ba33-682ed25b6d22" (UID: "8dc088f1-c679-4845-ba33-682ed25b6d22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:36:20.848690 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:20.847543 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dc088f1-c679-4845-ba33-682ed25b6d22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dc088f1-c679-4845-ba33-682ed25b6d22-workload-certs\") pod \"8dc088f1-c679-4845-ba33-682ed25b6d22\" (UID: \"8dc088f1-c679-4845-ba33-682ed25b6d22\") " Nov 03 14:36:20.848690 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:20.847730 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dc088f1-c679-4845-ba33-682ed25b6d22-istio-token\" (UniqueName: \"kubernetes.io/projected/8dc088f1-c679-4845-ba33-682ed25b6d22-istio-token\") pod \"8dc088f1-c679-4845-ba33-682ed25b6d22\" (UID: \"8dc088f1-c679-4845-ba33-682ed25b6d22\") " Nov 03 14:36:20.848690 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:20.847756 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dc088f1-c679-4845-ba33-682ed25b6d22-workload-certs" pod "8dc088f1-c679-4845-ba33-682ed25b6d22" (UID: "8dc088f1-c679-4845-ba33-682ed25b6d22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:36:20.848690 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:20.847766 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dc088f1-c679-4845-ba33-682ed25b6d22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dc088f1-c679-4845-ba33-682ed25b6d22-istiod-ca-cert\") pod \"8dc088f1-c679-4845-ba33-682ed25b6d22\" (UID: \"8dc088f1-c679-4845-ba33-682ed25b6d22\") " Nov 03 14:36:20.848690 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:20.847916 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dc088f1-c679-4845-ba33-682ed25b6d22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dc088f1-c679-4845-ba33-682ed25b6d22-workload-socket\") pod \"8dc088f1-c679-4845-ba33-682ed25b6d22\" (UID: \"8dc088f1-c679-4845-ba33-682ed25b6d22\") " Nov 03 14:36:20.848690 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:20.848369 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dc088f1-c679-4845-ba33-682ed25b6d22-istiod-ca-cert" pod "8dc088f1-c679-4845-ba33-682ed25b6d22" (UID: "8dc088f1-c679-4845-ba33-682ed25b6d22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:36:20.848690 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:20.848596 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dc088f1-c679-4845-ba33-682ed25b6d22-workload-socket" pod "8dc088f1-c679-4845-ba33-682ed25b6d22" (UID: "8dc088f1-c679-4845-ba33-682ed25b6d22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:36:20.848690 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:20.848667 5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dc088f1-c679-4845-ba33-682ed25b6d22-istio-data\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:36:20.848690 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:20.848688 5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dc088f1-c679-4845-ba33-682ed25b6d22-credential-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:36:20.849329 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:20.848705 5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dc088f1-c679-4845-ba33-682ed25b6d22-workload-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:36:20.849329 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:20.848733 5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dc088f1-c679-4845-ba33-682ed25b6d22-istiod-ca-cert\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:36:20.849329 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:20.849219 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dc088f1-c679-4845-ba33-682ed25b6d22-kube-api-access-8gdsq" pod "8dc088f1-c679-4845-ba33-682ed25b6d22" (UID: "8dc088f1-c679-4845-ba33-682ed25b6d22"). InnerVolumeSpecName "kube-api-access-8gdsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:36:20.849454 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:20.849368 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8dc088f1-c679-4845-ba33-682ed25b6d22-istio-podinfo" pod "8dc088f1-c679-4845-ba33-682ed25b6d22" (UID: "8dc088f1-c679-4845-ba33-682ed25b6d22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:36:20.849878 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:20.849828 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dc088f1-c679-4845-ba33-682ed25b6d22-istio-envoy" pod "8dc088f1-c679-4845-ba33-682ed25b6d22" (UID: "8dc088f1-c679-4845-ba33-682ed25b6d22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:36:20.850968 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:20.850944 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dc088f1-c679-4845-ba33-682ed25b6d22-istio-token" pod "8dc088f1-c679-4845-ba33-682ed25b6d22" (UID: "8dc088f1-c679-4845-ba33-682ed25b6d22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:36:20.949893 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:20.949848 5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dc088f1-c679-4845-ba33-682ed25b6d22-istio-token\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:36:20.949893 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:20.949892 5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dc088f1-c679-4845-ba33-682ed25b6d22-workload-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:36:20.949893 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:20.949906 5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dc088f1-c679-4845-ba33-682ed25b6d22-istio-podinfo\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:36:20.950127 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:20.949919 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gdsq\" (UniqueName: \"kubernetes.io/projected/8dc088f1-c679-4845-ba33-682ed25b6d22-kube-api-access-8gdsq\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:36:20.950127 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:20.949931 5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dc088f1-c679-4845-ba33-682ed25b6d22-istio-envoy\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:36:21.108817 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:21.108781 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc"] Nov 03 14:36:21.113961 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:21.113934 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-cb587c-openshift-default-6f8dff86cc-vhlqc"] Nov 03 14:36:22.251387 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:22.251358 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dc088f1-c679-4845-ba33-682ed25b6d22" path="/var/lib/kubelet/pods/8dc088f1-c679-4845-ba33-682ed25b6d22/volumes" Nov 03 14:36:26.034859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:25.964327 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf"] Nov 03 14:36:26.117864 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:26.069167 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" podUID="7ae02904-4e94-487e-a422-2749cdb51c4f" containerName="istio-proxy" containerID="cri-o://97f6add77a8a520b3597edfd97deb350f1c92448a8be6f96e1dae411cc8c10f4" gracePeriod=30 Nov 03 14:36:31.471244 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:31.471208 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" Nov 03 14:36:31.511516 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:31.511483 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ae02904-4e94-487e-a422-2749cdb51c4f-istio-token\" (UniqueName: \"kubernetes.io/projected/7ae02904-4e94-487e-a422-2749cdb51c4f-istio-token\") pod \"7ae02904-4e94-487e-a422-2749cdb51c4f\" (UID: \"7ae02904-4e94-487e-a422-2749cdb51c4f\") " Nov 03 14:36:31.511674 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:31.511585 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ae02904-4e94-487e-a422-2749cdb51c4f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ae02904-4e94-487e-a422-2749cdb51c4f-workload-certs\") pod \"7ae02904-4e94-487e-a422-2749cdb51c4f\" (UID: \"7ae02904-4e94-487e-a422-2749cdb51c4f\") " Nov 03 14:36:31.511674 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:31.511652 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ae02904-4e94-487e-a422-2749cdb51c4f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae02904-4e94-487e-a422-2749cdb51c4f-credential-socket\") pod \"7ae02904-4e94-487e-a422-2749cdb51c4f\" (UID: \"7ae02904-4e94-487e-a422-2749cdb51c4f\") " Nov 03 14:36:31.511794 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:31.511687 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ae02904-4e94-487e-a422-2749cdb51c4f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ae02904-4e94-487e-a422-2749cdb51c4f-istio-podinfo\") pod \"7ae02904-4e94-487e-a422-2749cdb51c4f\" (UID: \"7ae02904-4e94-487e-a422-2749cdb51c4f\") " Nov 03 14:36:31.511794 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:31.511716 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ae02904-4e94-487e-a422-2749cdb51c4f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ae02904-4e94-487e-a422-2749cdb51c4f-istiod-ca-cert\") pod \"7ae02904-4e94-487e-a422-2749cdb51c4f\" (UID: \"7ae02904-4e94-487e-a422-2749cdb51c4f\") " Nov 03 14:36:31.511940 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:31.511910 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ae02904-4e94-487e-a422-2749cdb51c4f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ae02904-4e94-487e-a422-2749cdb51c4f-istio-envoy\") pod \"7ae02904-4e94-487e-a422-2749cdb51c4f\" (UID: \"7ae02904-4e94-487e-a422-2749cdb51c4f\") " Nov 03 14:36:31.512042 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:31.511927 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ae02904-4e94-487e-a422-2749cdb51c4f-workload-certs" pod "7ae02904-4e94-487e-a422-2749cdb51c4f" (UID: "7ae02904-4e94-487e-a422-2749cdb51c4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:36:31.512042 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:31.511944 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ae02904-4e94-487e-a422-2749cdb51c4f-kube-api-access-7nv8h\" (UniqueName: \"kubernetes.io/projected/7ae02904-4e94-487e-a422-2749cdb51c4f-kube-api-access-7nv8h\") pod \"7ae02904-4e94-487e-a422-2749cdb51c4f\" (UID: \"7ae02904-4e94-487e-a422-2749cdb51c4f\") " Nov 03 14:36:31.512042 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:31.511984 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ae02904-4e94-487e-a422-2749cdb51c4f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ae02904-4e94-487e-a422-2749cdb51c4f-istio-data\") pod \"7ae02904-4e94-487e-a422-2749cdb51c4f\" (UID: \"7ae02904-4e94-487e-a422-2749cdb51c4f\") " Nov 03 14:36:31.512042 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:31.512014 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ae02904-4e94-487e-a422-2749cdb51c4f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae02904-4e94-487e-a422-2749cdb51c4f-workload-socket\") pod \"7ae02904-4e94-487e-a422-2749cdb51c4f\" (UID: \"7ae02904-4e94-487e-a422-2749cdb51c4f\") " Nov 03 14:36:31.512236 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:31.511978 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ae02904-4e94-487e-a422-2749cdb51c4f-credential-socket" pod "7ae02904-4e94-487e-a422-2749cdb51c4f" (UID: "7ae02904-4e94-487e-a422-2749cdb51c4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:36:31.512236 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:31.512172 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ae02904-4e94-487e-a422-2749cdb51c4f-istio-data" pod "7ae02904-4e94-487e-a422-2749cdb51c4f" (UID: "7ae02904-4e94-487e-a422-2749cdb51c4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:36:31.512342 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:31.512239 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ae02904-4e94-487e-a422-2749cdb51c4f-istiod-ca-cert" pod "7ae02904-4e94-487e-a422-2749cdb51c4f" (UID: "7ae02904-4e94-487e-a422-2749cdb51c4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:36:31.512396 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:31.512351 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ae02904-4e94-487e-a422-2749cdb51c4f-workload-socket" pod "7ae02904-4e94-487e-a422-2749cdb51c4f" (UID: "7ae02904-4e94-487e-a422-2749cdb51c4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:36:31.512718 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:31.512693 5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ae02904-4e94-487e-a422-2749cdb51c4f-istio-data\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:36:31.512818 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:31.512722 5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae02904-4e94-487e-a422-2749cdb51c4f-workload-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:36:31.512818 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:31.512738 5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ae02904-4e94-487e-a422-2749cdb51c4f-workload-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:36:31.512818 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:31.512753 5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae02904-4e94-487e-a422-2749cdb51c4f-credential-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:36:31.512818 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:31.512768 5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ae02904-4e94-487e-a422-2749cdb51c4f-istiod-ca-cert\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:36:31.514129 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:31.514106 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ae02904-4e94-487e-a422-2749cdb51c4f-istio-token" pod "7ae02904-4e94-487e-a422-2749cdb51c4f" (UID: "7ae02904-4e94-487e-a422-2749cdb51c4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:36:31.514251 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:31.514183 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ae02904-4e94-487e-a422-2749cdb51c4f-istio-envoy" pod "7ae02904-4e94-487e-a422-2749cdb51c4f" (UID: "7ae02904-4e94-487e-a422-2749cdb51c4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:36:31.514462 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:31.514434 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ae02904-4e94-487e-a422-2749cdb51c4f-istio-podinfo" pod "7ae02904-4e94-487e-a422-2749cdb51c4f" (UID: "7ae02904-4e94-487e-a422-2749cdb51c4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:36:31.514559 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:31.514469 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ae02904-4e94-487e-a422-2749cdb51c4f-kube-api-access-7nv8h" pod "7ae02904-4e94-487e-a422-2749cdb51c4f" (UID: "7ae02904-4e94-487e-a422-2749cdb51c4f"). InnerVolumeSpecName "kube-api-access-7nv8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:36:31.614423 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:31.614386 5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ae02904-4e94-487e-a422-2749cdb51c4f-istio-envoy\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:36:31.614423 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:31.614420 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nv8h\" (UniqueName: \"kubernetes.io/projected/7ae02904-4e94-487e-a422-2749cdb51c4f-kube-api-access-7nv8h\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:36:31.614423 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:31.614432 5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ae02904-4e94-487e-a422-2749cdb51c4f-istio-token\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:36:31.614742 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:31.614440 5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ae02904-4e94-487e-a422-2749cdb51c4f-istio-podinfo\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:36:32.038403 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.038352 5060 generic.go:356] "Generic (PLEG): container finished" podID="7ae02904-4e94-487e-a422-2749cdb51c4f" containerID="97f6add77a8a520b3597edfd97deb350f1c92448a8be6f96e1dae411cc8c10f4" exitCode=0 Nov 03 14:36:32.038604 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.038416 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" Nov 03 14:36:32.038604 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.038425 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" event={"ID":"7ae02904-4e94-487e-a422-2749cdb51c4f","Type":"ContainerDied","Data":"97f6add77a8a520b3597edfd97deb350f1c92448a8be6f96e1dae411cc8c10f4"} Nov 03 14:36:32.038604 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.038497 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf" event={"ID":"7ae02904-4e94-487e-a422-2749cdb51c4f","Type":"ContainerDied","Data":"688057c779c84db33fe71f5c0c38710f3249d0ec515615a4438385d9ac8587f7"} Nov 03 14:36:32.038604 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.038511 5060 scope.go:122] "RemoveContainer" containerID="97f6add77a8a520b3597edfd97deb350f1c92448a8be6f96e1dae411cc8c10f4" Nov 03 14:36:32.051046 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.051028 5060 scope.go:122] "RemoveContainer" containerID="083147e066d5126568ef8097c11067f2637dd8df409af907b3d6793beb16add6" Nov 03 14:36:32.063572 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.063550 5060 scope.go:122] "RemoveContainer" containerID="97f6add77a8a520b3597edfd97deb350f1c92448a8be6f96e1dae411cc8c10f4" Nov 03 14:36:32.064100 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:36:32.064068 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97f6add77a8a520b3597edfd97deb350f1c92448a8be6f96e1dae411cc8c10f4\": container with ID starting with 97f6add77a8a520b3597edfd97deb350f1c92448a8be6f96e1dae411cc8c10f4 not found: ID does not exist" containerID="97f6add77a8a520b3597edfd97deb350f1c92448a8be6f96e1dae411cc8c10f4" Nov 03 14:36:32.064196 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.064109 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97f6add77a8a520b3597edfd97deb350f1c92448a8be6f96e1dae411cc8c10f4"} err="failed to get container status \"97f6add77a8a520b3597edfd97deb350f1c92448a8be6f96e1dae411cc8c10f4\": rpc error: code = NotFound desc = could not find container \"97f6add77a8a520b3597edfd97deb350f1c92448a8be6f96e1dae411cc8c10f4\": container with ID starting with 97f6add77a8a520b3597edfd97deb350f1c92448a8be6f96e1dae411cc8c10f4 not found: ID does not exist" Nov 03 14:36:32.064196 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.064130 5060 scope.go:122] "RemoveContainer" containerID="083147e066d5126568ef8097c11067f2637dd8df409af907b3d6793beb16add6" Nov 03 14:36:32.064648 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:36:32.064620 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"083147e066d5126568ef8097c11067f2637dd8df409af907b3d6793beb16add6\": container with ID starting with 083147e066d5126568ef8097c11067f2637dd8df409af907b3d6793beb16add6 not found: ID does not exist" containerID="083147e066d5126568ef8097c11067f2637dd8df409af907b3d6793beb16add6" Nov 03 14:36:32.064748 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.064654 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"083147e066d5126568ef8097c11067f2637dd8df409af907b3d6793beb16add6"} err="failed to get container status \"083147e066d5126568ef8097c11067f2637dd8df409af907b3d6793beb16add6\": rpc error: code = NotFound desc = could not find container \"083147e066d5126568ef8097c11067f2637dd8df409af907b3d6793beb16add6\": container with ID starting with 083147e066d5126568ef8097c11067f2637dd8df409af907b3d6793beb16add6 not found: ID does not exist" Nov 03 14:36:32.114885 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.114852 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf"] Nov 03 14:36:32.118873 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.118820 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-4a0349-openshift-default-5b8ff4f965-28xzf"] Nov 03 14:36:32.154175 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.154140 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-682a07/noten-model-682a07-kserve-7484b86d8f-wwl9q"] Nov 03 14:36:32.155038 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.155016 5060 cpu_manager.go:414] "containerMap: removing container" podUID="3dc0b682-756a-4736-9d3a-ab20195b4bcb" containerName="main" Nov 03 14:36:32.155221 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.155193 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dc0b682-756a-4736-9d3a-ab20195b4bcb" containerName="main" Nov 03 14:36:32.155347 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.155234 5060 cpu_manager.go:414] "containerMap: removing container" podUID="7ae02904-4e94-487e-a422-2749cdb51c4f" containerName="istio-proxy" Nov 03 14:36:32.155347 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.155243 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ae02904-4e94-487e-a422-2749cdb51c4f" containerName="istio-proxy" Nov 03 14:36:32.155347 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.155268 5060 cpu_manager.go:414] "containerMap: removing container" podUID="8dc088f1-c679-4845-ba33-682ed25b6d22" containerName="istio-proxy" Nov 03 14:36:32.155347 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.155276 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dc088f1-c679-4845-ba33-682ed25b6d22" containerName="istio-proxy" Nov 03 14:36:32.155347 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.155290 5060 cpu_manager.go:414] "containerMap: removing container" podUID="b93ca36c-db33-4a19-b384-331987d1bb9f" containerName="maas-api" Nov 03 14:36:32.155347 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.155297 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b93ca36c-db33-4a19-b384-331987d1bb9f" containerName="maas-api" Nov 03 14:36:32.155347 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.155314 5060 cpu_manager.go:414] "containerMap: removing container" podUID="8dc088f1-c679-4845-ba33-682ed25b6d22" containerName="istio-proxy" Nov 03 14:36:32.155347 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.155321 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dc088f1-c679-4845-ba33-682ed25b6d22" containerName="istio-proxy" Nov 03 14:36:32.155347 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.155340 5060 cpu_manager.go:414] "containerMap: removing container" podUID="7ae02904-4e94-487e-a422-2749cdb51c4f" containerName="istio-proxy" Nov 03 14:36:32.155347 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.155348 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ae02904-4e94-487e-a422-2749cdb51c4f" containerName="istio-proxy" Nov 03 14:36:32.155866 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.155519 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dc0b682-756a-4736-9d3a-ab20195b4bcb" containerName="main" Nov 03 14:36:32.155866 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.155538 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ae02904-4e94-487e-a422-2749cdb51c4f" containerName="istio-proxy" Nov 03 14:36:32.155866 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.155548 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dc088f1-c679-4845-ba33-682ed25b6d22" containerName="istio-proxy" Nov 03 14:36:32.155866 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.155560 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="b93ca36c-db33-4a19-b384-331987d1bb9f" containerName="maas-api" Nov 03 14:36:32.155866 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.155568 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dc088f1-c679-4845-ba33-682ed25b6d22" containerName="istio-proxy" Nov 03 14:36:32.161065 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.161044 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-682a07/noten-model-682a07-kserve-7484b86d8f-wwl9q" Nov 03 14:36:32.178366 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.178325 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-682a07\"/\"default-dockercfg-hm6gb\"" Nov 03 14:36:32.180682 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.180657 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-682a07/noten-model-682a07-kserve-7484b86d8f-wwl9q"] Nov 03 14:36:32.181075 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.181052 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-682a07\"/\"noten-model-682a07-kserve-self-signed-certs\"" Nov 03 14:36:32.181075 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.181055 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-682a07\"/\"openshift-service-ca.crt\"" Nov 03 14:36:32.181075 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.181055 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-682a07\"/\"kube-root-ca.crt\"" Nov 03 14:36:32.222339 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.222280 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/754a16e2-2dde-45b9-a87e-c70c9e3db26e-tmp-dir\") pod \"noten-model-682a07-kserve-7484b86d8f-wwl9q\" (UID: \"754a16e2-2dde-45b9-a87e-c70c9e3db26e\") " pod="ai-tenant-e2e-noten-682a07/noten-model-682a07-kserve-7484b86d8f-wwl9q" Nov 03 14:36:32.222576 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.222415 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/754a16e2-2dde-45b9-a87e-c70c9e3db26e-home\") pod \"noten-model-682a07-kserve-7484b86d8f-wwl9q\" (UID: \"754a16e2-2dde-45b9-a87e-c70c9e3db26e\") " pod="ai-tenant-e2e-noten-682a07/noten-model-682a07-kserve-7484b86d8f-wwl9q" Nov 03 14:36:32.222576 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.222535 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/754a16e2-2dde-45b9-a87e-c70c9e3db26e-model-cache\") pod \"noten-model-682a07-kserve-7484b86d8f-wwl9q\" (UID: \"754a16e2-2dde-45b9-a87e-c70c9e3db26e\") " pod="ai-tenant-e2e-noten-682a07/noten-model-682a07-kserve-7484b86d8f-wwl9q" Nov 03 14:36:32.222688 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.222574 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/754a16e2-2dde-45b9-a87e-c70c9e3db26e-tls-certs\") pod \"noten-model-682a07-kserve-7484b86d8f-wwl9q\" (UID: \"754a16e2-2dde-45b9-a87e-c70c9e3db26e\") " pod="ai-tenant-e2e-noten-682a07/noten-model-682a07-kserve-7484b86d8f-wwl9q" Nov 03 14:36:32.222688 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.222616 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rpg4\" (UniqueName: \"kubernetes.io/projected/754a16e2-2dde-45b9-a87e-c70c9e3db26e-kube-api-access-8rpg4\") pod \"noten-model-682a07-kserve-7484b86d8f-wwl9q\" (UID: \"754a16e2-2dde-45b9-a87e-c70c9e3db26e\") " pod="ai-tenant-e2e-noten-682a07/noten-model-682a07-kserve-7484b86d8f-wwl9q" Nov 03 14:36:32.222688 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.222645 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/754a16e2-2dde-45b9-a87e-c70c9e3db26e-dshm\") pod \"noten-model-682a07-kserve-7484b86d8f-wwl9q\" (UID: \"754a16e2-2dde-45b9-a87e-c70c9e3db26e\") " pod="ai-tenant-e2e-noten-682a07/noten-model-682a07-kserve-7484b86d8f-wwl9q" Nov 03 14:36:32.251447 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.251412 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ae02904-4e94-487e-a422-2749cdb51c4f" path="/var/lib/kubelet/pods/7ae02904-4e94-487e-a422-2749cdb51c4f/volumes" Nov 03 14:36:32.305167 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.305077 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl"] Nov 03 14:36:32.305341 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.305317 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl" podUID="232c1666-ad86-40b0-b7e5-3da4d4d00200" containerName="maas-api" containerID="cri-o://b4d3564916fb4a0dfd65688d8c1fe352f0a896af24ae09b4648f448e3183d79a" gracePeriod=30 Nov 03 14:36:32.324322 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.324297 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/754a16e2-2dde-45b9-a87e-c70c9e3db26e-tmp-dir\") pod \"noten-model-682a07-kserve-7484b86d8f-wwl9q\" (UID: \"754a16e2-2dde-45b9-a87e-c70c9e3db26e\") " pod="ai-tenant-e2e-noten-682a07/noten-model-682a07-kserve-7484b86d8f-wwl9q" Nov 03 14:36:32.324537 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.324518 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/754a16e2-2dde-45b9-a87e-c70c9e3db26e-home\") pod \"noten-model-682a07-kserve-7484b86d8f-wwl9q\" (UID: \"754a16e2-2dde-45b9-a87e-c70c9e3db26e\") " pod="ai-tenant-e2e-noten-682a07/noten-model-682a07-kserve-7484b86d8f-wwl9q" Nov 03 14:36:32.326208 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.324721 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/754a16e2-2dde-45b9-a87e-c70c9e3db26e-model-cache\") pod \"noten-model-682a07-kserve-7484b86d8f-wwl9q\" (UID: \"754a16e2-2dde-45b9-a87e-c70c9e3db26e\") " pod="ai-tenant-e2e-noten-682a07/noten-model-682a07-kserve-7484b86d8f-wwl9q" Nov 03 14:36:32.326208 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.324771 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/754a16e2-2dde-45b9-a87e-c70c9e3db26e-tls-certs\") pod \"noten-model-682a07-kserve-7484b86d8f-wwl9q\" (UID: \"754a16e2-2dde-45b9-a87e-c70c9e3db26e\") " pod="ai-tenant-e2e-noten-682a07/noten-model-682a07-kserve-7484b86d8f-wwl9q" Nov 03 14:36:32.326208 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.324813 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rpg4\" (UniqueName: \"kubernetes.io/projected/754a16e2-2dde-45b9-a87e-c70c9e3db26e-kube-api-access-8rpg4\") pod \"noten-model-682a07-kserve-7484b86d8f-wwl9q\" (UID: \"754a16e2-2dde-45b9-a87e-c70c9e3db26e\") " pod="ai-tenant-e2e-noten-682a07/noten-model-682a07-kserve-7484b86d8f-wwl9q" Nov 03 14:36:32.326208 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.324874 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/754a16e2-2dde-45b9-a87e-c70c9e3db26e-dshm\") pod \"noten-model-682a07-kserve-7484b86d8f-wwl9q\" (UID: \"754a16e2-2dde-45b9-a87e-c70c9e3db26e\") " pod="ai-tenant-e2e-noten-682a07/noten-model-682a07-kserve-7484b86d8f-wwl9q" Nov 03 14:36:32.326208 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.325387 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/754a16e2-2dde-45b9-a87e-c70c9e3db26e-tmp-dir\") pod \"noten-model-682a07-kserve-7484b86d8f-wwl9q\" (UID: \"754a16e2-2dde-45b9-a87e-c70c9e3db26e\") " pod="ai-tenant-e2e-noten-682a07/noten-model-682a07-kserve-7484b86d8f-wwl9q" Nov 03 14:36:32.326208 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.325622 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/754a16e2-2dde-45b9-a87e-c70c9e3db26e-home\") pod \"noten-model-682a07-kserve-7484b86d8f-wwl9q\" (UID: \"754a16e2-2dde-45b9-a87e-c70c9e3db26e\") " pod="ai-tenant-e2e-noten-682a07/noten-model-682a07-kserve-7484b86d8f-wwl9q" Nov 03 14:36:32.326208 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.326138 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/754a16e2-2dde-45b9-a87e-c70c9e3db26e-model-cache\") pod \"noten-model-682a07-kserve-7484b86d8f-wwl9q\" (UID: \"754a16e2-2dde-45b9-a87e-c70c9e3db26e\") " pod="ai-tenant-e2e-noten-682a07/noten-model-682a07-kserve-7484b86d8f-wwl9q" Nov 03 14:36:32.330344 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.330124 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/754a16e2-2dde-45b9-a87e-c70c9e3db26e-dshm\") pod \"noten-model-682a07-kserve-7484b86d8f-wwl9q\" (UID: \"754a16e2-2dde-45b9-a87e-c70c9e3db26e\") " pod="ai-tenant-e2e-noten-682a07/noten-model-682a07-kserve-7484b86d8f-wwl9q" Nov 03 14:36:32.332975 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.332955 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/754a16e2-2dde-45b9-a87e-c70c9e3db26e-tls-certs\") pod \"noten-model-682a07-kserve-7484b86d8f-wwl9q\" (UID: \"754a16e2-2dde-45b9-a87e-c70c9e3db26e\") " pod="ai-tenant-e2e-noten-682a07/noten-model-682a07-kserve-7484b86d8f-wwl9q" Nov 03 14:36:32.358976 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.358923 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rpg4\" (UniqueName: \"kubernetes.io/projected/754a16e2-2dde-45b9-a87e-c70c9e3db26e-kube-api-access-8rpg4\") pod \"noten-model-682a07-kserve-7484b86d8f-wwl9q\" (UID: \"754a16e2-2dde-45b9-a87e-c70c9e3db26e\") " pod="ai-tenant-e2e-noten-682a07/noten-model-682a07-kserve-7484b86d8f-wwl9q" Nov 03 14:36:32.475924 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.475815 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-682a07/noten-model-682a07-kserve-7484b86d8f-wwl9q" Nov 03 14:36:32.825656 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:32.825625 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl" Nov 03 14:36:33.003085 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:33.003054 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/232c1666-ad86-40b0-b7e5-3da4d4d00200-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/232c1666-ad86-40b0-b7e5-3da4d4d00200-openshift-service-ca\") pod \"232c1666-ad86-40b0-b7e5-3da4d4d00200\" (UID: \"232c1666-ad86-40b0-b7e5-3da4d4d00200\") " Nov 03 14:36:33.003085 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:33.003097 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/232c1666-ad86-40b0-b7e5-3da4d4d00200-kube-api-access-dfzhw\" (UniqueName: \"kubernetes.io/projected/232c1666-ad86-40b0-b7e5-3da4d4d00200-kube-api-access-dfzhw\") pod \"232c1666-ad86-40b0-b7e5-3da4d4d00200\" (UID: \"232c1666-ad86-40b0-b7e5-3da4d4d00200\") " Nov 03 14:36:33.003462 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:33.003437 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/232c1666-ad86-40b0-b7e5-3da4d4d00200-metrics-certs\" (UniqueName: \"kubernetes.io/secret/232c1666-ad86-40b0-b7e5-3da4d4d00200-metrics-certs\") pod \"232c1666-ad86-40b0-b7e5-3da4d4d00200\" (UID: \"232c1666-ad86-40b0-b7e5-3da4d4d00200\") " Nov 03 14:36:33.003462 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:33.003450 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/232c1666-ad86-40b0-b7e5-3da4d4d00200-openshift-service-ca" pod "232c1666-ad86-40b0-b7e5-3da4d4d00200" (UID: "232c1666-ad86-40b0-b7e5-3da4d4d00200"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:36:33.003604 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:33.003476 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/232c1666-ad86-40b0-b7e5-3da4d4d00200-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/232c1666-ad86-40b0-b7e5-3da4d4d00200-maas-api-tls\") pod \"232c1666-ad86-40b0-b7e5-3da4d4d00200\" (UID: \"232c1666-ad86-40b0-b7e5-3da4d4d00200\") " Nov 03 14:36:33.004084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:33.004061 5060 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/232c1666-ad86-40b0-b7e5-3da4d4d00200-openshift-service-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:36:33.005727 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:33.005706 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/232c1666-ad86-40b0-b7e5-3da4d4d00200-metrics-certs" pod "232c1666-ad86-40b0-b7e5-3da4d4d00200" (UID: "232c1666-ad86-40b0-b7e5-3da4d4d00200"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:36:33.005868 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:33.005755 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/232c1666-ad86-40b0-b7e5-3da4d4d00200-kube-api-access-dfzhw" pod "232c1666-ad86-40b0-b7e5-3da4d4d00200" (UID: "232c1666-ad86-40b0-b7e5-3da4d4d00200"). InnerVolumeSpecName "kube-api-access-dfzhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:36:33.006046 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:33.006031 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/232c1666-ad86-40b0-b7e5-3da4d4d00200-maas-api-tls" pod "232c1666-ad86-40b0-b7e5-3da4d4d00200" (UID: "232c1666-ad86-40b0-b7e5-3da4d4d00200"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:36:33.049038 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:33.049001 5060 generic.go:356] "Generic (PLEG): container finished" podID="232c1666-ad86-40b0-b7e5-3da4d4d00200" containerID="b4d3564916fb4a0dfd65688d8c1fe352f0a896af24ae09b4648f448e3183d79a" exitCode=0 Nov 03 14:36:33.049218 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:33.049060 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl" Nov 03 14:36:33.049218 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:33.049090 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl" event={"ID":"232c1666-ad86-40b0-b7e5-3da4d4d00200","Type":"ContainerDied","Data":"b4d3564916fb4a0dfd65688d8c1fe352f0a896af24ae09b4648f448e3183d79a"} Nov 03 14:36:33.049218 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:33.049152 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl" event={"ID":"232c1666-ad86-40b0-b7e5-3da4d4d00200","Type":"ContainerDied","Data":"686e3e73b9a40077ced240d1cdfe26933fc32842739fa5c134d693510e88ff2a"} Nov 03 14:36:33.049218 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:33.049178 5060 scope.go:122] "RemoveContainer" containerID="b4d3564916fb4a0dfd65688d8c1fe352f0a896af24ae09b4648f448e3183d79a" Nov 03 14:36:33.051380 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:33.051349 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-682a07/noten-model-682a07-kserve-7484b86d8f-wwl9q"] Nov 03 14:36:33.053164 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:36:33.053141 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod754a16e2_2dde_45b9_a87e_c70c9e3db26e.slice/crio-612a37b4a64570b1faf212d3727105b632f1356ff670558472ed6801482b4695 WatchSource:0}: Error finding container 612a37b4a64570b1faf212d3727105b632f1356ff670558472ed6801482b4695: Status 404 returned error can't find the container with id 612a37b4a64570b1faf212d3727105b632f1356ff670558472ed6801482b4695 Nov 03 14:36:33.060531 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:33.060515 5060 scope.go:122] "RemoveContainer" containerID="b4d3564916fb4a0dfd65688d8c1fe352f0a896af24ae09b4648f448e3183d79a" Nov 03 14:36:33.060784 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:36:33.060758 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4d3564916fb4a0dfd65688d8c1fe352f0a896af24ae09b4648f448e3183d79a\": container with ID starting with b4d3564916fb4a0dfd65688d8c1fe352f0a896af24ae09b4648f448e3183d79a not found: ID does not exist" containerID="b4d3564916fb4a0dfd65688d8c1fe352f0a896af24ae09b4648f448e3183d79a" Nov 03 14:36:33.060892 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:33.060794 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4d3564916fb4a0dfd65688d8c1fe352f0a896af24ae09b4648f448e3183d79a"} err="failed to get container status \"b4d3564916fb4a0dfd65688d8c1fe352f0a896af24ae09b4648f448e3183d79a\": rpc error: code = NotFound desc = could not find container \"b4d3564916fb4a0dfd65688d8c1fe352f0a896af24ae09b4648f448e3183d79a\": container with ID starting with b4d3564916fb4a0dfd65688d8c1fe352f0a896af24ae09b4648f448e3183d79a not found: ID does not exist" Nov 03 14:36:33.108438 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:33.106184 5060 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/232c1666-ad86-40b0-b7e5-3da4d4d00200-metrics-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:36:33.108438 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:33.106218 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/232c1666-ad86-40b0-b7e5-3da4d4d00200-maas-api-tls\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:36:33.108438 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:33.106232 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfzhw\" (UniqueName: \"kubernetes.io/projected/232c1666-ad86-40b0-b7e5-3da4d4d00200-kube-api-access-dfzhw\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:36:33.120858 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:33.120777 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl"] Nov 03 14:36:33.123511 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:33.123484 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2e8614-7dfb7d5788-676fl"] Nov 03 14:36:34.060024 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:34.059991 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-682a07/noten-model-682a07-kserve-7484b86d8f-wwl9q" event={"ID":"754a16e2-2dde-45b9-a87e-c70c9e3db26e","Type":"ContainerStarted","Data":"17bcc1267ccd38c48492487ceac0b0989fb869993a4803e7150906568770f036"} Nov 03 14:36:34.060024 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:34.060031 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-682a07/noten-model-682a07-kserve-7484b86d8f-wwl9q" event={"ID":"754a16e2-2dde-45b9-a87e-c70c9e3db26e","Type":"ContainerStarted","Data":"612a37b4a64570b1faf212d3727105b632f1356ff670558472ed6801482b4695"} Nov 03 14:36:34.110860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:34.110785 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-682a07/noten-model-682a07-kserve-7484b86d8f-wwl9q" podStartSLOduration=2.110767459 podStartE2EDuration="2.110767459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:36:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:36:34.106827176 +0000 UTC m=+1527.995616758" watchObservedRunningTime="2026-11-03 14:36:34.110767459 +0000 UTC m=+1527.999557041" Nov 03 14:36:34.257967 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:34.257932 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="232c1666-ad86-40b0-b7e5-3da4d4d00200" path="/var/lib/kubelet/pods/232c1666-ad86-40b0-b7e5-3da4d4d00200/volumes" Nov 03 14:36:34.476397 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:34.476317 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-682a07/noten-model-682a07-kserve-7484b86d8f-wwl9q" Nov 03 14:36:39.440768 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:36:39.440722 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:36:42.476084 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:42.476049 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-682a07/noten-model-682a07-kserve-7484b86d8f-wwl9q" Nov 03 14:36:42.489271 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:42.489244 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-682a07/noten-model-682a07-kserve-7484b86d8f-wwl9q" Nov 03 14:36:43.143117 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:43.143087 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-682a07/noten-model-682a07-kserve-7484b86d8f-wwl9q" Nov 03 14:36:47.205232 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:47.205180 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-682a07/noten-model-682a07-kserve-7484b86d8f-wwl9q"] Nov 03 14:36:59.613014 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:59.612963 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:36:59.613732 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:59.613681 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" podUID="9737ed5d-be9c-45ef-afd6-977eddcc3ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:36:59.896919 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:36:59.895919 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/distinct-2-w3-156ae1-kserve-5988894d76-vsdlm" podUID="59dcdef3-f88e-4463-a0dd-1917e915a516" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 14:37:00.189497 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:00.189401 5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-156ae1/distinct-2-w3-156ae1-kserve-5988894d76-vsdlm" podUID="59dcdef3-f88e-4463-a0dd-1917e915a516" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded" Nov 03 14:37:02.239366 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:02.239300 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-682a07/noten-model-682a07-kserve-7484b86d8f-wwl9q" podUID="754a16e2-2dde-45b9-a87e-c70c9e3db26e" containerName="main" containerID="cri-o://17bcc1267ccd38c48492487ceac0b0989fb869993a4803e7150906568770f036" gracePeriod=45 Nov 03 14:37:02.537109 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:02.537078 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-682a07/noten-model-682a07-kserve-7484b86d8f-wwl9q" Nov 03 14:37:02.691723 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:02.691685 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/754a16e2-2dde-45b9-a87e-c70c9e3db26e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/754a16e2-2dde-45b9-a87e-c70c9e3db26e-model-cache\") pod \"754a16e2-2dde-45b9-a87e-c70c9e3db26e\" (UID: \"754a16e2-2dde-45b9-a87e-c70c9e3db26e\") " Nov 03 14:37:02.691931 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:02.691808 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/754a16e2-2dde-45b9-a87e-c70c9e3db26e-kube-api-access-8rpg4\" (UniqueName: \"kubernetes.io/projected/754a16e2-2dde-45b9-a87e-c70c9e3db26e-kube-api-access-8rpg4\") pod \"754a16e2-2dde-45b9-a87e-c70c9e3db26e\" (UID: \"754a16e2-2dde-45b9-a87e-c70c9e3db26e\") " Nov 03 14:37:02.691931 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:02.691897 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/754a16e2-2dde-45b9-a87e-c70c9e3db26e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/754a16e2-2dde-45b9-a87e-c70c9e3db26e-tmp-dir\") pod \"754a16e2-2dde-45b9-a87e-c70c9e3db26e\" (UID: \"754a16e2-2dde-45b9-a87e-c70c9e3db26e\") " Nov 03 14:37:02.692072 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:02.691961 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/754a16e2-2dde-45b9-a87e-c70c9e3db26e-tls-certs\" (UniqueName: \"kubernetes.io/secret/754a16e2-2dde-45b9-a87e-c70c9e3db26e-tls-certs\") pod \"754a16e2-2dde-45b9-a87e-c70c9e3db26e\" (UID: \"754a16e2-2dde-45b9-a87e-c70c9e3db26e\") " Nov 03 14:37:02.692072 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:02.691981 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/754a16e2-2dde-45b9-a87e-c70c9e3db26e-home\" (UniqueName: \"kubernetes.io/empty-dir/754a16e2-2dde-45b9-a87e-c70c9e3db26e-home\") pod \"754a16e2-2dde-45b9-a87e-c70c9e3db26e\" (UID: \"754a16e2-2dde-45b9-a87e-c70c9e3db26e\") " Nov 03 14:37:02.692072 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:02.692008 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/754a16e2-2dde-45b9-a87e-c70c9e3db26e-model-cache" pod "754a16e2-2dde-45b9-a87e-c70c9e3db26e" (UID: "754a16e2-2dde-45b9-a87e-c70c9e3db26e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:37:02.692072 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:02.692042 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/754a16e2-2dde-45b9-a87e-c70c9e3db26e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/754a16e2-2dde-45b9-a87e-c70c9e3db26e-dshm\") pod \"754a16e2-2dde-45b9-a87e-c70c9e3db26e\" (UID: \"754a16e2-2dde-45b9-a87e-c70c9e3db26e\") " Nov 03 14:37:02.692289 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:02.692207 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/754a16e2-2dde-45b9-a87e-c70c9e3db26e-tmp-dir" pod "754a16e2-2dde-45b9-a87e-c70c9e3db26e" (UID: "754a16e2-2dde-45b9-a87e-c70c9e3db26e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:37:02.692289 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:02.692247 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/754a16e2-2dde-45b9-a87e-c70c9e3db26e-home" pod "754a16e2-2dde-45b9-a87e-c70c9e3db26e" (UID: "754a16e2-2dde-45b9-a87e-c70c9e3db26e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:37:02.692518 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:02.692501 5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/754a16e2-2dde-45b9-a87e-c70c9e3db26e-model-cache\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:37:02.692585 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:02.692520 5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/754a16e2-2dde-45b9-a87e-c70c9e3db26e-tmp-dir\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:37:02.692585 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:02.692528 5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/754a16e2-2dde-45b9-a87e-c70c9e3db26e-home\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:37:02.694210 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:02.694180 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/754a16e2-2dde-45b9-a87e-c70c9e3db26e-dshm" pod "754a16e2-2dde-45b9-a87e-c70c9e3db26e" (UID: "754a16e2-2dde-45b9-a87e-c70c9e3db26e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:37:02.694210 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:02.694199 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/754a16e2-2dde-45b9-a87e-c70c9e3db26e-kube-api-access-8rpg4" pod "754a16e2-2dde-45b9-a87e-c70c9e3db26e" (UID: "754a16e2-2dde-45b9-a87e-c70c9e3db26e"). InnerVolumeSpecName "kube-api-access-8rpg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:37:02.694210 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:02.694189 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/754a16e2-2dde-45b9-a87e-c70c9e3db26e-tls-certs" pod "754a16e2-2dde-45b9-a87e-c70c9e3db26e" (UID: "754a16e2-2dde-45b9-a87e-c70c9e3db26e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:37:02.794523 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:02.794441 5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/754a16e2-2dde-45b9-a87e-c70c9e3db26e-dshm\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:37:02.794523 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:02.794482 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rpg4\" (UniqueName: \"kubernetes.io/projected/754a16e2-2dde-45b9-a87e-c70c9e3db26e-kube-api-access-8rpg4\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:37:02.794523 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:02.794498 5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/754a16e2-2dde-45b9-a87e-c70c9e3db26e-tls-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:37:03.228122 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.228035 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt"] Nov 03 14:37:03.228876 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.228857 5060 cpu_manager.go:414] "containerMap: removing container" podUID="232c1666-ad86-40b0-b7e5-3da4d4d00200" containerName="maas-api" Nov 03 14:37:03.228932 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.228879 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="232c1666-ad86-40b0-b7e5-3da4d4d00200" containerName="maas-api" Nov 03 14:37:03.228975 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.228964 5060 cpu_manager.go:414] "containerMap: removing container" podUID="754a16e2-2dde-45b9-a87e-c70c9e3db26e" containerName="main" Nov 03 14:37:03.229014 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.228976 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="754a16e2-2dde-45b9-a87e-c70c9e3db26e" containerName="main" Nov 03 14:37:03.229119 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.229107 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ae02904-4e94-487e-a422-2749cdb51c4f" containerName="istio-proxy" Nov 03 14:37:03.229152 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.229123 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="754a16e2-2dde-45b9-a87e-c70c9e3db26e" containerName="main" Nov 03 14:37:03.229152 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.229137 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="232c1666-ad86-40b0-b7e5-3da4d4d00200" containerName="maas-api" Nov 03 14:37:03.233269 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.233248 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" Nov 03 14:37:03.266371 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.266341 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-331c9e-openshift-default-dockercfg-skzpv\"" Nov 03 14:37:03.271455 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.271424 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt"] Nov 03 14:37:03.348107 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.348073 5060 generic.go:356] "Generic (PLEG): container finished" podID="754a16e2-2dde-45b9-a87e-c70c9e3db26e" containerID="17bcc1267ccd38c48492487ceac0b0989fb869993a4803e7150906568770f036" exitCode=0 Nov 03 14:37:03.348309 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.348117 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-682a07/noten-model-682a07-kserve-7484b86d8f-wwl9q" event={"ID":"754a16e2-2dde-45b9-a87e-c70c9e3db26e","Type":"ContainerDied","Data":"17bcc1267ccd38c48492487ceac0b0989fb869993a4803e7150906568770f036"} Nov 03 14:37:03.348309 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.348162 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-682a07/noten-model-682a07-kserve-7484b86d8f-wwl9q" event={"ID":"754a16e2-2dde-45b9-a87e-c70c9e3db26e","Type":"ContainerDied","Data":"612a37b4a64570b1faf212d3727105b632f1356ff670558472ed6801482b4695"} Nov 03 14:37:03.348309 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.348185 5060 scope.go:122] "RemoveContainer" containerID="17bcc1267ccd38c48492487ceac0b0989fb869993a4803e7150906568770f036" Nov 03 14:37:03.348309 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.348125 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-682a07/noten-model-682a07-kserve-7484b86d8f-wwl9q" Nov 03 14:37:03.358021 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.357994 5060 scope.go:122] "RemoveContainer" containerID="17bcc1267ccd38c48492487ceac0b0989fb869993a4803e7150906568770f036" Nov 03 14:37:03.358301 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:37:03.358282 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17bcc1267ccd38c48492487ceac0b0989fb869993a4803e7150906568770f036\": container with ID starting with 17bcc1267ccd38c48492487ceac0b0989fb869993a4803e7150906568770f036 not found: ID does not exist" containerID="17bcc1267ccd38c48492487ceac0b0989fb869993a4803e7150906568770f036" Nov 03 14:37:03.358358 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.358309 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17bcc1267ccd38c48492487ceac0b0989fb869993a4803e7150906568770f036"} err="failed to get container status \"17bcc1267ccd38c48492487ceac0b0989fb869993a4803e7150906568770f036\": rpc error: code = NotFound desc = could not find container \"17bcc1267ccd38c48492487ceac0b0989fb869993a4803e7150906568770f036\": container with ID starting with 17bcc1267ccd38c48492487ceac0b0989fb869993a4803e7150906568770f036 not found: ID does not exist" Nov 03 14:37:03.401998 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.401963 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0653ff5-b9b7-4ed9-8311-df75f12109ca-credential-socket\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt\" (UID: \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" Nov 03 14:37:03.402164 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.402007 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0653ff5-b9b7-4ed9-8311-df75f12109ca-istio-data\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt\" (UID: \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" Nov 03 14:37:03.402164 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.402092 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0653ff5-b9b7-4ed9-8311-df75f12109ca-istiod-ca-cert\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt\" (UID: \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" Nov 03 14:37:03.402164 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.402123 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0653ff5-b9b7-4ed9-8311-df75f12109ca-workload-certs\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt\" (UID: \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" Nov 03 14:37:03.402164 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.402150 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0653ff5-b9b7-4ed9-8311-df75f12109ca-istio-token\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt\" (UID: \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" Nov 03 14:37:03.402315 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.402201 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0653ff5-b9b7-4ed9-8311-df75f12109ca-istio-envoy\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt\" (UID: \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" Nov 03 14:37:03.402315 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.402286 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0653ff5-b9b7-4ed9-8311-df75f12109ca-workload-socket\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt\" (UID: \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" Nov 03 14:37:03.402396 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.402334 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfs2l\" (UniqueName: \"kubernetes.io/projected/c0653ff5-b9b7-4ed9-8311-df75f12109ca-kube-api-access-nfs2l\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt\" (UID: \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" Nov 03 14:37:03.402472 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.402444 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0653ff5-b9b7-4ed9-8311-df75f12109ca-istio-podinfo\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt\" (UID: \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" Nov 03 14:37:03.407266 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.407240 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-682a07/noten-model-682a07-kserve-7484b86d8f-wwl9q"] Nov 03 14:37:03.414878 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.414853 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-682a07/noten-model-682a07-kserve-7484b86d8f-wwl9q"] Nov 03 14:37:03.504143 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.504101 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0653ff5-b9b7-4ed9-8311-df75f12109ca-credential-socket\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt\" (UID: \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" Nov 03 14:37:03.504352 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.504158 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0653ff5-b9b7-4ed9-8311-df75f12109ca-istio-data\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt\" (UID: \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" Nov 03 14:37:03.504352 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.504198 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0653ff5-b9b7-4ed9-8311-df75f12109ca-istiod-ca-cert\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt\" (UID: \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" Nov 03 14:37:03.504352 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.504229 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0653ff5-b9b7-4ed9-8311-df75f12109ca-workload-certs\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt\" (UID: \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" Nov 03 14:37:03.504556 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.504377 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0653ff5-b9b7-4ed9-8311-df75f12109ca-istio-token\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt\" (UID: \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" Nov 03 14:37:03.504556 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.504423 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0653ff5-b9b7-4ed9-8311-df75f12109ca-istio-envoy\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt\" (UID: \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" Nov 03 14:37:03.504556 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.504500 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0653ff5-b9b7-4ed9-8311-df75f12109ca-workload-socket\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt\" (UID: \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" Nov 03 14:37:03.504715 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.504575 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfs2l\" (UniqueName: \"kubernetes.io/projected/c0653ff5-b9b7-4ed9-8311-df75f12109ca-kube-api-access-nfs2l\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt\" (UID: \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" Nov 03 14:37:03.504715 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.504598 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0653ff5-b9b7-4ed9-8311-df75f12109ca-credential-socket\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt\" (UID: \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" Nov 03 14:37:03.504715 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.504634 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0653ff5-b9b7-4ed9-8311-df75f12109ca-istio-podinfo\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt\" (UID: \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" Nov 03 14:37:03.505102 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.505077 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0653ff5-b9b7-4ed9-8311-df75f12109ca-istiod-ca-cert\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt\" (UID: \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" Nov 03 14:37:03.505270 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.505241 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0653ff5-b9b7-4ed9-8311-df75f12109ca-istio-data\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt\" (UID: \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" Nov 03 14:37:03.505405 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.505331 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0653ff5-b9b7-4ed9-8311-df75f12109ca-workload-socket\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt\" (UID: \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" Nov 03 14:37:03.505557 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.505536 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0653ff5-b9b7-4ed9-8311-df75f12109ca-workload-certs\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt\" (UID: \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" Nov 03 14:37:03.507257 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.507227 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0653ff5-b9b7-4ed9-8311-df75f12109ca-istio-envoy\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt\" (UID: \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" Nov 03 14:37:03.507369 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.507286 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0653ff5-b9b7-4ed9-8311-df75f12109ca-istio-podinfo\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt\" (UID: \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" Nov 03 14:37:03.527790 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.527762 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0653ff5-b9b7-4ed9-8311-df75f12109ca-istio-token\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt\" (UID: \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" Nov 03 14:37:03.528055 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.528028 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfs2l\" (UniqueName: \"kubernetes.io/projected/c0653ff5-b9b7-4ed9-8311-df75f12109ca-kube-api-access-nfs2l\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt\" (UID: \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" Nov 03 14:37:03.548175 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.548142 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" Nov 03 14:37:03.869778 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.869745 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt"] Nov 03 14:37:03.871760 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:37:03.871731 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0653ff5_b9b7_4ed9_8311_df75f12109ca.slice/crio-75036d10805cf2fb507efcf916e8e896b1dec096c0dbb5e4300eee0dfbd7e047 WatchSource:0}: Error finding container 75036d10805cf2fb507efcf916e8e896b1dec096c0dbb5e4300eee0dfbd7e047: Status 404 returned error can't find the container with id 75036d10805cf2fb507efcf916e8e896b1dec096c0dbb5e4300eee0dfbd7e047 Nov 03 14:37:03.874394 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.874356 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:37:03.874475 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.874449 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:37:03.874516 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:03.874504 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:37:04.249428 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:04.249386 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="754a16e2-2dde-45b9-a87e-c70c9e3db26e" path="/var/lib/kubelet/pods/754a16e2-2dde-45b9-a87e-c70c9e3db26e/volumes" Nov 03 14:37:04.445864 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:04.445804 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" event={"ID":"c0653ff5-b9b7-4ed9-8311-df75f12109ca","Type":"ContainerStarted","Data":"f7043cf8764bec87135a994490e9f26507a7fb93c88374f2e16b6cb3745a2585"} Nov 03 14:37:04.446509 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:04.446480 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" event={"ID":"c0653ff5-b9b7-4ed9-8311-df75f12109ca","Type":"ContainerStarted","Data":"75036d10805cf2fb507efcf916e8e896b1dec096c0dbb5e4300eee0dfbd7e047"} Nov 03 14:37:04.549283 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:04.549193 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" Nov 03 14:37:04.550718 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:04.550679 5060 patch_prober.go:29] interesting pod/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 14:37:04.550939 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:04.550915 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" podUID="c0653ff5-b9b7-4ed9-8311-df75f12109ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 14:37:04.552623 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:04.552553 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" podStartSLOduration=1.552536866 podStartE2EDuration="1.552536866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:37:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:37:04.538349223 +0000 UTC m=+1558.427138831" watchObservedRunningTime="2026-11-03 14:37:04.552536866 +0000 UTC m=+1558.441326448" Nov 03 14:37:05.552290 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:05.552242 5060 patch_prober.go:29] interesting pod/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 14:37:05.552917 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:05.552867 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" podUID="c0653ff5-b9b7-4ed9-8311-df75f12109ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 14:37:06.549867 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:06.549235 5060 patch_prober.go:29] interesting pod/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 14:37:06.549867 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:06.549314 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" podUID="c0653ff5-b9b7-4ed9-8311-df75f12109ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 14:37:07.551253 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:07.551213 5060 patch_prober.go:29] interesting pod/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 14:37:07.551876 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:07.551824 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" podUID="c0653ff5-b9b7-4ed9-8311-df75f12109ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 14:37:07.744116 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:07.744079 5060 scope.go:122] "RemoveContainer" containerID="e93523379f2673a0a82ee63f0e98a9eb05421f8a51f76748b13b7eab52582b0a" Nov 03 14:37:07.817297 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:07.817267 5060 scope.go:122] "RemoveContainer" containerID="e3f004809bbde156e9f9fc103f8190afa70ca5983b1a3fe3464ee3dc91f04c09" Nov 03 14:37:08.548889 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:08.548854 5060 patch_prober.go:29] interesting pod/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 14:37:08.549622 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:08.549591 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" podUID="c0653ff5-b9b7-4ed9-8311-df75f12109ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 14:37:09.432858 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:09.431920 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98" podUID="3aaaeea4-134f-4d17-a180-b4857c6425d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 14:37:09.531858 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:09.530941 5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98" podUID="3aaaeea4-134f-4d17-a180-b4857c6425d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:37:09.563300 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:09.563259 5060 patch_prober.go:29] interesting pod/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 14:37:09.563577 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:09.563541 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" podUID="c0653ff5-b9b7-4ed9-8311-df75f12109ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 14:37:09.724003 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:37:09.720067 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:37:10.550273 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:10.550232 5060 patch_prober.go:29] interesting pod/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 14:37:10.550987 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:10.550947 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" podUID="c0653ff5-b9b7-4ed9-8311-df75f12109ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 14:37:11.570557 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:11.570503 5060 patch_prober.go:29] interesting pod/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 14:37:11.571582 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:11.571190 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" podUID="c0653ff5-b9b7-4ed9-8311-df75f12109ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 14:37:12.553482 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:12.553450 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" Nov 03 14:37:12.553735 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:12.553716 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" Nov 03 14:37:12.554358 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:12.554337 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" Nov 03 14:37:34.047242 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:34.047189 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1af1f14-zgc6j"] Nov 03 14:37:34.051437 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:34.051415 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1af1f14-zgc6j" Nov 03 14:37:34.074484 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:34.074451 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1af1f14-zgc6j"] Nov 03 14:37:34.082368 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:34.082127 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:37:34.082368 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:34.082175 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:37:34.082368 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:34.082278 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\"" Nov 03 14:37:34.082368 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:34.082339 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:37:34.137461 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:34.137425 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9jf7\" (UniqueName: \"kubernetes.io/projected/c29e0a5b-36c0-402f-90ca-b8c8fca88acd-kube-api-access-z9jf7\") pod \"maas-api-revoke-sub-f1af1f14-zgc6j\" (UID: \"c29e0a5b-36c0-402f-90ca-b8c8fca88acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1af1f14-zgc6j" Nov 03 14:37:34.137650 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:34.137480 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c29e0a5b-36c0-402f-90ca-b8c8fca88acd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1af1f14-zgc6j\" (UID: \"c29e0a5b-36c0-402f-90ca-b8c8fca88acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1af1f14-zgc6j" Nov 03 14:37:34.137778 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:34.137712 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c29e0a5b-36c0-402f-90ca-b8c8fca88acd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1af1f14-zgc6j\" (UID: \"c29e0a5b-36c0-402f-90ca-b8c8fca88acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1af1f14-zgc6j" Nov 03 14:37:34.238460 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:34.238431 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c29e0a5b-36c0-402f-90ca-b8c8fca88acd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1af1f14-zgc6j\" (UID: \"c29e0a5b-36c0-402f-90ca-b8c8fca88acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1af1f14-zgc6j" Nov 03 14:37:34.238628 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:34.238490 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9jf7\" (UniqueName: \"kubernetes.io/projected/c29e0a5b-36c0-402f-90ca-b8c8fca88acd-kube-api-access-z9jf7\") pod \"maas-api-revoke-sub-f1af1f14-zgc6j\" (UID: \"c29e0a5b-36c0-402f-90ca-b8c8fca88acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1af1f14-zgc6j" Nov 03 14:37:34.238628 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:34.238530 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c29e0a5b-36c0-402f-90ca-b8c8fca88acd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1af1f14-zgc6j\" (UID: \"c29e0a5b-36c0-402f-90ca-b8c8fca88acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1af1f14-zgc6j" Nov 03 14:37:34.239258 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:34.239237 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c29e0a5b-36c0-402f-90ca-b8c8fca88acd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1af1f14-zgc6j\" (UID: \"c29e0a5b-36c0-402f-90ca-b8c8fca88acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1af1f14-zgc6j" Nov 03 14:37:34.240810 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:34.240791 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c29e0a5b-36c0-402f-90ca-b8c8fca88acd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1af1f14-zgc6j\" (UID: \"c29e0a5b-36c0-402f-90ca-b8c8fca88acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1af1f14-zgc6j" Nov 03 14:37:34.260491 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:34.260460 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9jf7\" (UniqueName: \"kubernetes.io/projected/c29e0a5b-36c0-402f-90ca-b8c8fca88acd-kube-api-access-z9jf7\") pod \"maas-api-revoke-sub-f1af1f14-zgc6j\" (UID: \"c29e0a5b-36c0-402f-90ca-b8c8fca88acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1af1f14-zgc6j" Nov 03 14:37:34.366154 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:34.366066 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1af1f14-zgc6j" Nov 03 14:37:34.676995 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:34.676965 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1af1f14-zgc6j"] Nov 03 14:37:34.678080 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:37:34.678051 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc29e0a5b_36c0_402f_90ca_b8c8fca88acd.slice/crio-1608bb60eccb5b1848a718f172155f6fbb589615fdb7e1832efe7609e166dd8f WatchSource:0}: Error finding container 1608bb60eccb5b1848a718f172155f6fbb589615fdb7e1832efe7609e166dd8f: Status 404 returned error can't find the container with id 1608bb60eccb5b1848a718f172155f6fbb589615fdb7e1832efe7609e166dd8f Nov 03 14:37:34.822185 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:34.822149 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1af1f14-zgc6j" event={"ID":"c29e0a5b-36c0-402f-90ca-b8c8fca88acd","Type":"ContainerStarted","Data":"1608bb60eccb5b1848a718f172155f6fbb589615fdb7e1832efe7609e166dd8f"} Nov 03 14:37:35.829353 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:35.829318 5060 generic.go:356] "Generic (PLEG): container finished" podID="c29e0a5b-36c0-402f-90ca-b8c8fca88acd" containerID="3e8e3e65c8f572c1bd58abeae46e2b68afe56d34eb0d2a8aac8f8e9ec8a10f5c" exitCode=0 Nov 03 14:37:35.829733 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:35.829378 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1af1f14-zgc6j" event={"ID":"c29e0a5b-36c0-402f-90ca-b8c8fca88acd","Type":"ContainerDied","Data":"3e8e3e65c8f572c1bd58abeae46e2b68afe56d34eb0d2a8aac8f8e9ec8a10f5c"} Nov 03 14:37:36.996964 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:36.996933 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1af1f14-zgc6j" Nov 03 14:37:37.073768 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:37.073734 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c29e0a5b-36c0-402f-90ca-b8c8fca88acd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c29e0a5b-36c0-402f-90ca-b8c8fca88acd-maas-api-openshift-ca\") pod \"c29e0a5b-36c0-402f-90ca-b8c8fca88acd\" (UID: \"c29e0a5b-36c0-402f-90ca-b8c8fca88acd\") " Nov 03 14:37:37.073986 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:37.073804 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c29e0a5b-36c0-402f-90ca-b8c8fca88acd-kube-api-access-z9jf7\" (UniqueName: \"kubernetes.io/projected/c29e0a5b-36c0-402f-90ca-b8c8fca88acd-kube-api-access-z9jf7\") pod \"c29e0a5b-36c0-402f-90ca-b8c8fca88acd\" (UID: \"c29e0a5b-36c0-402f-90ca-b8c8fca88acd\") " Nov 03 14:37:37.073986 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:37.073830 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c29e0a5b-36c0-402f-90ca-b8c8fca88acd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c29e0a5b-36c0-402f-90ca-b8c8fca88acd-maas-api-xks-ca\") pod \"c29e0a5b-36c0-402f-90ca-b8c8fca88acd\" (UID: \"c29e0a5b-36c0-402f-90ca-b8c8fca88acd\") " Nov 03 14:37:37.074221 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:37.074190 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c29e0a5b-36c0-402f-90ca-b8c8fca88acd-maas-api-openshift-ca" pod "c29e0a5b-36c0-402f-90ca-b8c8fca88acd" (UID: "c29e0a5b-36c0-402f-90ca-b8c8fca88acd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:37:37.074636 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:37.074606 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c29e0a5b-36c0-402f-90ca-b8c8fca88acd-maas-api-openshift-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:37:37.076520 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:37.076494 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c29e0a5b-36c0-402f-90ca-b8c8fca88acd-kube-api-access-z9jf7" pod "c29e0a5b-36c0-402f-90ca-b8c8fca88acd" (UID: "c29e0a5b-36c0-402f-90ca-b8c8fca88acd"). InnerVolumeSpecName "kube-api-access-z9jf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:37:37.076635 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:37.076494 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c29e0a5b-36c0-402f-90ca-b8c8fca88acd-maas-api-xks-ca" pod "c29e0a5b-36c0-402f-90ca-b8c8fca88acd" (UID: "c29e0a5b-36c0-402f-90ca-b8c8fca88acd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:37:37.175782 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:37.175749 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9jf7\" (UniqueName: \"kubernetes.io/projected/c29e0a5b-36c0-402f-90ca-b8c8fca88acd-kube-api-access-z9jf7\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:37:37.175782 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:37.175778 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c29e0a5b-36c0-402f-90ca-b8c8fca88acd-maas-api-xks-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:37:37.844879 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:37.844815 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1af1f14-zgc6j" Nov 03 14:37:37.845050 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:37.844846 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1af1f14-zgc6j" event={"ID":"c29e0a5b-36c0-402f-90ca-b8c8fca88acd","Type":"ContainerDied","Data":"1608bb60eccb5b1848a718f172155f6fbb589615fdb7e1832efe7609e166dd8f"} Nov 03 14:37:37.845050 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:37.844972 5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1608bb60eccb5b1848a718f172155f6fbb589615fdb7e1832efe7609e166dd8f" Nov 03 14:37:39.442170 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:37:39.442123 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:37:48.514971 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:48.514881 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9040ed96-zkclk"] Nov 03 14:37:48.515761 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:48.515741 5060 cpu_manager.go:414] "containerMap: removing container" podUID="c29e0a5b-36c0-402f-90ca-b8c8fca88acd" containerName="revoke-keys" Nov 03 14:37:48.515819 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:48.515762 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c29e0a5b-36c0-402f-90ca-b8c8fca88acd" containerName="revoke-keys" Nov 03 14:37:48.515925 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:48.515913 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c29e0a5b-36c0-402f-90ca-b8c8fca88acd" containerName="revoke-keys" Nov 03 14:37:48.520086 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:48.520064 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9040ed96-zkclk" Nov 03 14:37:48.551625 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:48.551599 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:37:48.552112 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:48.551736 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\"" Nov 03 14:37:48.552112 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:48.552061 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:37:48.552112 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:48.552111 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:37:48.562950 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:48.559701 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9040ed96-zkclk"] Nov 03 14:37:48.690946 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:48.690910 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q47j5\" (UniqueName: \"kubernetes.io/projected/0cb8e7eb-3150-46e8-b9b6-48f53b265483-kube-api-access-q47j5\") pod \"maas-api-revoke-sub-9040ed96-zkclk\" (UID: \"0cb8e7eb-3150-46e8-b9b6-48f53b265483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9040ed96-zkclk" Nov 03 14:37:48.691155 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:48.691096 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cb8e7eb-3150-46e8-b9b6-48f53b265483-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9040ed96-zkclk\" (UID: \"0cb8e7eb-3150-46e8-b9b6-48f53b265483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9040ed96-zkclk" Nov 03 14:37:48.691232 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:48.691205 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cb8e7eb-3150-46e8-b9b6-48f53b265483-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9040ed96-zkclk\" (UID: \"0cb8e7eb-3150-46e8-b9b6-48f53b265483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9040ed96-zkclk" Nov 03 14:37:48.792478 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:48.792396 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cb8e7eb-3150-46e8-b9b6-48f53b265483-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9040ed96-zkclk\" (UID: \"0cb8e7eb-3150-46e8-b9b6-48f53b265483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9040ed96-zkclk" Nov 03 14:37:48.792478 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:48.792457 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cb8e7eb-3150-46e8-b9b6-48f53b265483-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9040ed96-zkclk\" (UID: \"0cb8e7eb-3150-46e8-b9b6-48f53b265483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9040ed96-zkclk" Nov 03 14:37:48.792705 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:48.792534 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q47j5\" (UniqueName: \"kubernetes.io/projected/0cb8e7eb-3150-46e8-b9b6-48f53b265483-kube-api-access-q47j5\") pod \"maas-api-revoke-sub-9040ed96-zkclk\" (UID: \"0cb8e7eb-3150-46e8-b9b6-48f53b265483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9040ed96-zkclk" Nov 03 14:37:48.793075 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:48.793048 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cb8e7eb-3150-46e8-b9b6-48f53b265483-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9040ed96-zkclk\" (UID: \"0cb8e7eb-3150-46e8-b9b6-48f53b265483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9040ed96-zkclk" Nov 03 14:37:48.794865 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:48.794812 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cb8e7eb-3150-46e8-b9b6-48f53b265483-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9040ed96-zkclk\" (UID: \"0cb8e7eb-3150-46e8-b9b6-48f53b265483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9040ed96-zkclk" Nov 03 14:37:48.814196 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:48.814172 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q47j5\" (UniqueName: \"kubernetes.io/projected/0cb8e7eb-3150-46e8-b9b6-48f53b265483-kube-api-access-q47j5\") pod \"maas-api-revoke-sub-9040ed96-zkclk\" (UID: \"0cb8e7eb-3150-46e8-b9b6-48f53b265483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9040ed96-zkclk" Nov 03 14:37:48.836939 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:48.836915 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9040ed96-zkclk" Nov 03 14:37:49.186359 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:49.186333 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9040ed96-zkclk"] Nov 03 14:37:49.188494 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:37:49.188463 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cb8e7eb_3150_46e8_b9b6_48f53b265483.slice/crio-dae9cd1228079bd38947f7b9fa177d6c46cf94abe1838803eb31dda647fe2ea4 WatchSource:0}: Error finding container dae9cd1228079bd38947f7b9fa177d6c46cf94abe1838803eb31dda647fe2ea4: Status 404 returned error can't find the container with id dae9cd1228079bd38947f7b9fa177d6c46cf94abe1838803eb31dda647fe2ea4 Nov 03 14:37:49.930907 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:49.930778 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9040ed96-zkclk" event={"ID":"0cb8e7eb-3150-46e8-b9b6-48f53b265483","Type":"ContainerStarted","Data":"40110fad18a3c3df8e51cfd8c284bf4d3ca618cf47cfec86462906504ab203c1"} Nov 03 14:37:49.930907 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:49.930867 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9040ed96-zkclk" event={"ID":"0cb8e7eb-3150-46e8-b9b6-48f53b265483","Type":"ContainerStarted","Data":"dae9cd1228079bd38947f7b9fa177d6c46cf94abe1838803eb31dda647fe2ea4"} Nov 03 14:37:49.979843 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:49.979769 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9040ed96-zkclk" podStartSLOduration=1.979750198 podStartE2EDuration="1.979750198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:37:49.977880974 +0000 UTC m=+1603.866670552" watchObservedRunningTime="2026-11-03 14:37:49.979750198 +0000 UTC m=+1603.868539781" Nov 03 14:37:50.950473 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:50.950438 5060 generic.go:356] "Generic (PLEG): container finished" podID="0cb8e7eb-3150-46e8-b9b6-48f53b265483" containerID="40110fad18a3c3df8e51cfd8c284bf4d3ca618cf47cfec86462906504ab203c1" exitCode=0 Nov 03 14:37:50.951172 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:50.951144 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9040ed96-zkclk" event={"ID":"0cb8e7eb-3150-46e8-b9b6-48f53b265483","Type":"ContainerDied","Data":"40110fad18a3c3df8e51cfd8c284bf4d3ca618cf47cfec86462906504ab203c1"} Nov 03 14:37:53.281577 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:53.281537 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9040ed96-zkclk" event={"ID":"0cb8e7eb-3150-46e8-b9b6-48f53b265483","Type":"ContainerDied","Data":"dae9cd1228079bd38947f7b9fa177d6c46cf94abe1838803eb31dda647fe2ea4"} Nov 03 14:37:53.286498 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:53.286471 5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dae9cd1228079bd38947f7b9fa177d6c46cf94abe1838803eb31dda647fe2ea4" Nov 03 14:37:53.296170 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:53.296146 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9040ed96-zkclk" Nov 03 14:37:53.412408 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:53.412351 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cb8e7eb-3150-46e8-b9b6-48f53b265483-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cb8e7eb-3150-46e8-b9b6-48f53b265483-maas-api-xks-ca\") pod \"0cb8e7eb-3150-46e8-b9b6-48f53b265483\" (UID: \"0cb8e7eb-3150-46e8-b9b6-48f53b265483\") " Nov 03 14:37:53.414199 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:53.414084 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cb8e7eb-3150-46e8-b9b6-48f53b265483-maas-api-openshift-ca" pod "0cb8e7eb-3150-46e8-b9b6-48f53b265483" (UID: "0cb8e7eb-3150-46e8-b9b6-48f53b265483"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:37:53.485722 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:53.412828 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cb8e7eb-3150-46e8-b9b6-48f53b265483-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cb8e7eb-3150-46e8-b9b6-48f53b265483-maas-api-openshift-ca\") pod \"0cb8e7eb-3150-46e8-b9b6-48f53b265483\" (UID: \"0cb8e7eb-3150-46e8-b9b6-48f53b265483\") " Nov 03 14:37:53.485722 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:53.484432 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cb8e7eb-3150-46e8-b9b6-48f53b265483-kube-api-access-q47j5\" (UniqueName: \"kubernetes.io/projected/0cb8e7eb-3150-46e8-b9b6-48f53b265483-kube-api-access-q47j5\") pod \"0cb8e7eb-3150-46e8-b9b6-48f53b265483\" (UID: \"0cb8e7eb-3150-46e8-b9b6-48f53b265483\") " Nov 03 14:37:53.485722 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:53.485274 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cb8e7eb-3150-46e8-b9b6-48f53b265483-maas-api-openshift-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:37:53.516194 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:53.516135 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cb8e7eb-3150-46e8-b9b6-48f53b265483-kube-api-access-q47j5" pod "0cb8e7eb-3150-46e8-b9b6-48f53b265483" (UID: "0cb8e7eb-3150-46e8-b9b6-48f53b265483"). InnerVolumeSpecName "kube-api-access-q47j5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:37:53.573907 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:53.560069 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cb8e7eb-3150-46e8-b9b6-48f53b265483-maas-api-xks-ca" pod "0cb8e7eb-3150-46e8-b9b6-48f53b265483" (UID: "0cb8e7eb-3150-46e8-b9b6-48f53b265483"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:37:53.743860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:53.641108 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q47j5\" (UniqueName: \"kubernetes.io/projected/0cb8e7eb-3150-46e8-b9b6-48f53b265483-kube-api-access-q47j5\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:37:53.743860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:53.641143 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cb8e7eb-3150-46e8-b9b6-48f53b265483-maas-api-xks-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:37:54.420856 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:37:54.420253 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9040ed96-zkclk" Nov 03 14:38:08.066487 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:08.066452 5060 scope.go:122] "RemoveContainer" containerID="8efab72153bc7b881341f7f0e7289e6e0d0ad4dd43e16bfa6669a3ca77ce78a5" Nov 03 14:38:09.547872 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:38:09.547063 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:38:13.242637 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:13.242598 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd"] Nov 03 14:38:13.243427 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:13.243392 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" podUID="6c8d7e93-3520-4a18-916a-ac77ae09f885" containerName="istio-proxy" containerID="cri-o://3feae28bb8fa9d9deb1e32ba02f68f4da3523e3787f19f09502d8ab81856ba5b" gracePeriod=30 Nov 03 14:38:13.280011 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:13.276340 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28"] Nov 03 14:38:13.280011 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:13.279146 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" podUID="2aafae9b-8aa6-46fe-bb81-a37c0e52fff7" containerName="istio-proxy" containerID="cri-o://e0aae71cc1e2efea9cb4773f11bae8882d915dd610406a7bf19bde889f1d44d2" gracePeriod=30 Nov 03 14:38:18.728422 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:18.728379 5060 generic.go:356] "Generic (PLEG): container finished" podID="6c8d7e93-3520-4a18-916a-ac77ae09f885" containerID="3feae28bb8fa9d9deb1e32ba02f68f4da3523e3787f19f09502d8ab81856ba5b" exitCode=0 Nov 03 14:38:18.728958 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:18.728510 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" event={"ID":"6c8d7e93-3520-4a18-916a-ac77ae09f885","Type":"ContainerDied","Data":"3feae28bb8fa9d9deb1e32ba02f68f4da3523e3787f19f09502d8ab81856ba5b"} Nov 03 14:38:18.728958 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:18.728567 5060 scope.go:122] "RemoveContainer" containerID="572d43f258e5370b50d58feb6d259d6c0b88b376677bad0e8cc855d1e9381b29" Nov 03 14:38:18.732590 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:18.732561 5060 generic.go:356] "Generic (PLEG): container finished" podID="2aafae9b-8aa6-46fe-bb81-a37c0e52fff7" containerID="e0aae71cc1e2efea9cb4773f11bae8882d915dd610406a7bf19bde889f1d44d2" exitCode=0 Nov 03 14:38:18.732880 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:18.732756 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" event={"ID":"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7","Type":"ContainerDied","Data":"e0aae71cc1e2efea9cb4773f11bae8882d915dd610406a7bf19bde889f1d44d2"} Nov 03 14:38:18.882242 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:18.881914 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" Nov 03 14:38:18.947178 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:18.947121 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c8d7e93-3520-4a18-916a-ac77ae09f885-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c8d7e93-3520-4a18-916a-ac77ae09f885-workload-certs\") pod \"6c8d7e93-3520-4a18-916a-ac77ae09f885\" (UID: \"6c8d7e93-3520-4a18-916a-ac77ae09f885\") " Nov 03 14:38:18.973571 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:18.973448 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c8d7e93-3520-4a18-916a-ac77ae09f885-workload-certs" pod "6c8d7e93-3520-4a18-916a-ac77ae09f885" (UID: "6c8d7e93-3520-4a18-916a-ac77ae09f885"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:18.974019 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:18.973966 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c8d7e93-3520-4a18-916a-ac77ae09f885-kube-api-access-gmc62\" (UniqueName: \"kubernetes.io/projected/6c8d7e93-3520-4a18-916a-ac77ae09f885-kube-api-access-gmc62\") pod \"6c8d7e93-3520-4a18-916a-ac77ae09f885\" (UID: \"6c8d7e93-3520-4a18-916a-ac77ae09f885\") " Nov 03 14:38:18.974225 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:18.974175 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c8d7e93-3520-4a18-916a-ac77ae09f885-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8d7e93-3520-4a18-916a-ac77ae09f885-credential-socket\") pod \"6c8d7e93-3520-4a18-916a-ac77ae09f885\" (UID: \"6c8d7e93-3520-4a18-916a-ac77ae09f885\") " Nov 03 14:38:18.983878 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:18.974358 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c8d7e93-3520-4a18-916a-ac77ae09f885-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c8d7e93-3520-4a18-916a-ac77ae09f885-istio-envoy\") pod \"6c8d7e93-3520-4a18-916a-ac77ae09f885\" (UID: \"6c8d7e93-3520-4a18-916a-ac77ae09f885\") " Nov 03 14:38:19.014877 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.006517 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c8d7e93-3520-4a18-916a-ac77ae09f885-kube-api-access-gmc62" pod "6c8d7e93-3520-4a18-916a-ac77ae09f885" (UID: "6c8d7e93-3520-4a18-916a-ac77ae09f885"). InnerVolumeSpecName "kube-api-access-gmc62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:19.047880 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.039022 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c8d7e93-3520-4a18-916a-ac77ae09f885-credential-socket" pod "6c8d7e93-3520-4a18-916a-ac77ae09f885" (UID: "6c8d7e93-3520-4a18-916a-ac77ae09f885"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:19.079910 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.067060 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c8d7e93-3520-4a18-916a-ac77ae09f885-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c8d7e93-3520-4a18-916a-ac77ae09f885-istiod-ca-cert\") pod \"6c8d7e93-3520-4a18-916a-ac77ae09f885\" (UID: \"6c8d7e93-3520-4a18-916a-ac77ae09f885\") " Nov 03 14:38:19.079910 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.067119 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c8d7e93-3520-4a18-916a-ac77ae09f885-istio-token\" (UniqueName: \"kubernetes.io/projected/6c8d7e93-3520-4a18-916a-ac77ae09f885-istio-token\") pod \"6c8d7e93-3520-4a18-916a-ac77ae09f885\" (UID: \"6c8d7e93-3520-4a18-916a-ac77ae09f885\") " Nov 03 14:38:19.079910 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.067189 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c8d7e93-3520-4a18-916a-ac77ae09f885-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8d7e93-3520-4a18-916a-ac77ae09f885-workload-socket\") pod \"6c8d7e93-3520-4a18-916a-ac77ae09f885\" (UID: \"6c8d7e93-3520-4a18-916a-ac77ae09f885\") " Nov 03 14:38:19.079910 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.067245 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c8d7e93-3520-4a18-916a-ac77ae09f885-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c8d7e93-3520-4a18-916a-ac77ae09f885-istio-data\") pod \"6c8d7e93-3520-4a18-916a-ac77ae09f885\" (UID: \"6c8d7e93-3520-4a18-916a-ac77ae09f885\") " Nov 03 14:38:19.079910 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.067289 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c8d7e93-3520-4a18-916a-ac77ae09f885-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c8d7e93-3520-4a18-916a-ac77ae09f885-istio-podinfo\") pod \"6c8d7e93-3520-4a18-916a-ac77ae09f885\" (UID: \"6c8d7e93-3520-4a18-916a-ac77ae09f885\") " Nov 03 14:38:19.079910 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.067952 5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c8d7e93-3520-4a18-916a-ac77ae09f885-workload-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:19.079910 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.067977 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmc62\" (UniqueName: \"kubernetes.io/projected/6c8d7e93-3520-4a18-916a-ac77ae09f885-kube-api-access-gmc62\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:19.079910 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.067993 5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8d7e93-3520-4a18-916a-ac77ae09f885-credential-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:19.116955 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.116200 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c8d7e93-3520-4a18-916a-ac77ae09f885-istio-data" pod "6c8d7e93-3520-4a18-916a-ac77ae09f885" (UID: "6c8d7e93-3520-4a18-916a-ac77ae09f885"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:19.116955 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.116547 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c8d7e93-3520-4a18-916a-ac77ae09f885-istiod-ca-cert" pod "6c8d7e93-3520-4a18-916a-ac77ae09f885" (UID: "6c8d7e93-3520-4a18-916a-ac77ae09f885"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:38:19.191962 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.117921 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c8d7e93-3520-4a18-916a-ac77ae09f885-workload-socket" pod "6c8d7e93-3520-4a18-916a-ac77ae09f885" (UID: "6c8d7e93-3520-4a18-916a-ac77ae09f885"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:19.191962 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.169173 5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c8d7e93-3520-4a18-916a-ac77ae09f885-istiod-ca-cert\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:19.191962 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.169207 5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8d7e93-3520-4a18-916a-ac77ae09f885-workload-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:19.191962 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.169238 5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c8d7e93-3520-4a18-916a-ac77ae09f885-istio-data\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:19.290857 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.289253 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c8d7e93-3520-4a18-916a-ac77ae09f885-istio-podinfo" pod "6c8d7e93-3520-4a18-916a-ac77ae09f885" (UID: "6c8d7e93-3520-4a18-916a-ac77ae09f885"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:38:19.290857 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.290769 5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c8d7e93-3520-4a18-916a-ac77ae09f885-istio-podinfo\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:19.295046 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.295011 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c8d7e93-3520-4a18-916a-ac77ae09f885-istio-token" pod "6c8d7e93-3520-4a18-916a-ac77ae09f885" (UID: "6c8d7e93-3520-4a18-916a-ac77ae09f885"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:19.379879 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.378204 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c8d7e93-3520-4a18-916a-ac77ae09f885-istio-envoy" pod "6c8d7e93-3520-4a18-916a-ac77ae09f885" (UID: "6c8d7e93-3520-4a18-916a-ac77ae09f885"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:19.391818 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.391759 5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c8d7e93-3520-4a18-916a-ac77ae09f885-istio-envoy\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:19.391818 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.391792 5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c8d7e93-3520-4a18-916a-ac77ae09f885-istio-token\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:19.436664 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.436366 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" Nov 03 14:38:19.492422 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.492387 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-credential-socket\") pod \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\" (UID: \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\") " Nov 03 14:38:19.493050 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.493019 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-istio-podinfo\") pod \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\" (UID: \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\") " Nov 03 14:38:19.493195 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.493130 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-workload-certs\") pod \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\" (UID: \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\") " Nov 03 14:38:19.493267 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.493198 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-istio-data\") pod \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\" (UID: \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\") " Nov 03 14:38:19.493328 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.493273 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-istiod-ca-cert\") pod \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\" (UID: \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\") " Nov 03 14:38:19.493393 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.493360 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-workload-socket\") pod \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\" (UID: \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\") " Nov 03 14:38:19.493447 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.493390 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-kube-api-access-wjd5p\" (UniqueName: \"kubernetes.io/projected/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-kube-api-access-wjd5p\") pod \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\" (UID: \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\") " Nov 03 14:38:19.493447 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.493435 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-istio-envoy\") pod \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\" (UID: \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\") " Nov 03 14:38:19.493557 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.493475 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-istio-token\" (UniqueName: \"kubernetes.io/projected/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-istio-token\") pod \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\" (UID: \"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7\") " Nov 03 14:38:19.494522 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.492882 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-credential-socket" pod "2aafae9b-8aa6-46fe-bb81-a37c0e52fff7" (UID: "2aafae9b-8aa6-46fe-bb81-a37c0e52fff7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:19.494637 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.494494 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-istio-data" pod "2aafae9b-8aa6-46fe-bb81-a37c0e52fff7" (UID: "2aafae9b-8aa6-46fe-bb81-a37c0e52fff7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:19.495255 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.495079 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-workload-socket" pod "2aafae9b-8aa6-46fe-bb81-a37c0e52fff7" (UID: "2aafae9b-8aa6-46fe-bb81-a37c0e52fff7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:19.495522 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.495500 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-istiod-ca-cert" pod "2aafae9b-8aa6-46fe-bb81-a37c0e52fff7" (UID: "2aafae9b-8aa6-46fe-bb81-a37c0e52fff7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:38:19.496594 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.496571 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-workload-certs" pod "2aafae9b-8aa6-46fe-bb81-a37c0e52fff7" (UID: "2aafae9b-8aa6-46fe-bb81-a37c0e52fff7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:19.519922 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.519889 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-kube-api-access-wjd5p" pod "2aafae9b-8aa6-46fe-bb81-a37c0e52fff7" (UID: "2aafae9b-8aa6-46fe-bb81-a37c0e52fff7"). InnerVolumeSpecName "kube-api-access-wjd5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:19.528019 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.527991 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-istio-envoy" pod "2aafae9b-8aa6-46fe-bb81-a37c0e52fff7" (UID: "2aafae9b-8aa6-46fe-bb81-a37c0e52fff7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:19.536304 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.536254 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-istio-podinfo" pod "2aafae9b-8aa6-46fe-bb81-a37c0e52fff7" (UID: "2aafae9b-8aa6-46fe-bb81-a37c0e52fff7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:38:19.538164 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.538130 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-istio-token" pod "2aafae9b-8aa6-46fe-bb81-a37c0e52fff7" (UID: "2aafae9b-8aa6-46fe-bb81-a37c0e52fff7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:19.595132 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.595105 5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-istio-podinfo\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:19.595132 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.595130 5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-workload-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:19.595319 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.595140 5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-istio-data\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:19.595319 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.595161 5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-istiod-ca-cert\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:19.595319 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.595171 5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-workload-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:19.595319 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.595182 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjd5p\" (UniqueName: \"kubernetes.io/projected/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-kube-api-access-wjd5p\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:19.595319 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.595196 5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-istio-envoy\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:19.595319 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.595204 5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-istio-token\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:19.595319 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.595211 5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7-credential-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:19.745930 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.742437 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" Nov 03 14:38:19.745930 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.742459 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd" event={"ID":"6c8d7e93-3520-4a18-916a-ac77ae09f885","Type":"ContainerDied","Data":"a6d3d89160dbd24b8c929373bddba0afe3be9d4cdc04ba27e58c8530285daba8"} Nov 03 14:38:19.745930 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.742549 5060 scope.go:122] "RemoveContainer" containerID="3feae28bb8fa9d9deb1e32ba02f68f4da3523e3787f19f09502d8ab81856ba5b" Nov 03 14:38:19.745930 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.744237 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" Nov 03 14:38:19.745930 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.744245 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28" event={"ID":"2aafae9b-8aa6-46fe-bb81-a37c0e52fff7","Type":"ContainerDied","Data":"519ff47f1e4a0878d3cf111b9bdc8fe63e8bea5dfe9853479684071c14987eba"} Nov 03 14:38:19.754527 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.754502 5060 scope.go:122] "RemoveContainer" containerID="e0aae71cc1e2efea9cb4773f11bae8882d915dd610406a7bf19bde889f1d44d2" Nov 03 14:38:19.809172 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.809142 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28"] Nov 03 14:38:19.813921 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.813895 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-tkq28"] Nov 03 14:38:19.842384 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.842357 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd"] Nov 03 14:38:19.850143 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:19.850113 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-9bnvd"] Nov 03 14:38:20.251533 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:20.251502 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aafae9b-8aa6-46fe-bb81-a37c0e52fff7" path="/var/lib/kubelet/pods/2aafae9b-8aa6-46fe-bb81-a37c0e52fff7/volumes" Nov 03 14:38:20.252016 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:20.252002 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c8d7e93-3520-4a18-916a-ac77ae09f885" path="/var/lib/kubelet/pods/6c8d7e93-3520-4a18-916a-ac77ae09f885/volumes" Nov 03 14:38:26.150286 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:26.150252 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5"] Nov 03 14:38:26.150807 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:26.150483 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" podUID="c8172c42-332c-40e4-9317-9014c5749253" containerName="istio-proxy" containerID="cri-o://a1519cd1e4d0fa11fa27ab35ab2f35597611cc362f25997eae15530a155c8208" gracePeriod=30 Nov 03 14:38:26.161249 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:26.161218 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg"] Nov 03 14:38:26.161449 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:26.161427 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" podUID="f859590f-c6cb-4b6c-bfd2-ebde80002e81" containerName="istio-proxy" containerID="cri-o://34b172cbeb89aca4f1e6ca29cce8e5cfad22ead649e482baf802a82bb6b1e173" gracePeriod=30 Nov 03 14:38:26.172294 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:26.172269 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc"] Nov 03 14:38:26.172706 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:26.172639 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" podUID="b671512d-59b5-4ddd-959d-1d9150d41268" containerName="istio-proxy" containerID="cri-o://5e9e8520dd24462be6b1e66bcad82e3f8d92ffebe4a2569073dd63c91f586a01" gracePeriod=30 Nov 03 14:38:31.471040 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.471006 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" Nov 03 14:38:31.522728 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.522702 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" Nov 03 14:38:31.525999 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.525966 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" Nov 03 14:38:31.561968 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.561934 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f859590f-c6cb-4b6c-bfd2-ebde80002e81-istio-token\" (UniqueName: \"kubernetes.io/projected/f859590f-c6cb-4b6c-bfd2-ebde80002e81-istio-token\") pod \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\" (UID: \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\") " Nov 03 14:38:31.562175 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.562050 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f859590f-c6cb-4b6c-bfd2-ebde80002e81-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f859590f-c6cb-4b6c-bfd2-ebde80002e81-istio-envoy\") pod \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\" (UID: \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\") " Nov 03 14:38:31.562175 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.562085 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f859590f-c6cb-4b6c-bfd2-ebde80002e81-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f859590f-c6cb-4b6c-bfd2-ebde80002e81-workload-certs\") pod \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\" (UID: \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\") " Nov 03 14:38:31.562175 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.562134 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f859590f-c6cb-4b6c-bfd2-ebde80002e81-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f859590f-c6cb-4b6c-bfd2-ebde80002e81-istio-data\") pod \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\" (UID: \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\") " Nov 03 14:38:31.562175 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.562170 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f859590f-c6cb-4b6c-bfd2-ebde80002e81-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f859590f-c6cb-4b6c-bfd2-ebde80002e81-istiod-ca-cert\") pod \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\" (UID: \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\") " Nov 03 14:38:31.562499 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.562220 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f859590f-c6cb-4b6c-bfd2-ebde80002e81-kube-api-access-8w2v7\" (UniqueName: \"kubernetes.io/projected/f859590f-c6cb-4b6c-bfd2-ebde80002e81-kube-api-access-8w2v7\") pod \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\" (UID: \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\") " Nov 03 14:38:31.562499 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.562308 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f859590f-c6cb-4b6c-bfd2-ebde80002e81-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f859590f-c6cb-4b6c-bfd2-ebde80002e81-istio-podinfo\") pod \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\" (UID: \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\") " Nov 03 14:38:31.562499 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.562396 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f859590f-c6cb-4b6c-bfd2-ebde80002e81-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f859590f-c6cb-4b6c-bfd2-ebde80002e81-credential-socket\") pod \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\" (UID: \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\") " Nov 03 14:38:31.562499 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.562428 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f859590f-c6cb-4b6c-bfd2-ebde80002e81-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f859590f-c6cb-4b6c-bfd2-ebde80002e81-workload-socket\") pod \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\" (UID: \"f859590f-c6cb-4b6c-bfd2-ebde80002e81\") " Nov 03 14:38:31.564596 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.564536 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f859590f-c6cb-4b6c-bfd2-ebde80002e81-credential-socket" pod "f859590f-c6cb-4b6c-bfd2-ebde80002e81" (UID: "f859590f-c6cb-4b6c-bfd2-ebde80002e81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:31.564890 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.564800 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f859590f-c6cb-4b6c-bfd2-ebde80002e81-workload-certs" pod "f859590f-c6cb-4b6c-bfd2-ebde80002e81" (UID: "f859590f-c6cb-4b6c-bfd2-ebde80002e81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:31.565364 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.565312 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f859590f-c6cb-4b6c-bfd2-ebde80002e81-istiod-ca-cert" pod "f859590f-c6cb-4b6c-bfd2-ebde80002e81" (UID: "f859590f-c6cb-4b6c-bfd2-ebde80002e81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:38:31.566150 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.566128 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f859590f-c6cb-4b6c-bfd2-ebde80002e81-istio-data" pod "f859590f-c6cb-4b6c-bfd2-ebde80002e81" (UID: "f859590f-c6cb-4b6c-bfd2-ebde80002e81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:31.566548 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.566526 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f859590f-c6cb-4b6c-bfd2-ebde80002e81-workload-socket" pod "f859590f-c6cb-4b6c-bfd2-ebde80002e81" (UID: "f859590f-c6cb-4b6c-bfd2-ebde80002e81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:31.568168 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.568141 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f859590f-c6cb-4b6c-bfd2-ebde80002e81-istio-envoy" pod "f859590f-c6cb-4b6c-bfd2-ebde80002e81" (UID: "f859590f-c6cb-4b6c-bfd2-ebde80002e81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:31.568310 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.568250 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f859590f-c6cb-4b6c-bfd2-ebde80002e81-kube-api-access-8w2v7" pod "f859590f-c6cb-4b6c-bfd2-ebde80002e81" (UID: "f859590f-c6cb-4b6c-bfd2-ebde80002e81"). InnerVolumeSpecName "kube-api-access-8w2v7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:31.568423 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.568389 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f859590f-c6cb-4b6c-bfd2-ebde80002e81-istio-token" pod "f859590f-c6cb-4b6c-bfd2-ebde80002e81" (UID: "f859590f-c6cb-4b6c-bfd2-ebde80002e81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:31.569799 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.569771 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f859590f-c6cb-4b6c-bfd2-ebde80002e81-istio-podinfo" pod "f859590f-c6cb-4b6c-bfd2-ebde80002e81" (UID: "f859590f-c6cb-4b6c-bfd2-ebde80002e81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:38:31.664446 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.664411 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8172c42-332c-40e4-9317-9014c5749253-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8172c42-332c-40e4-9317-9014c5749253-workload-socket\") pod \"c8172c42-332c-40e4-9317-9014c5749253\" (UID: \"c8172c42-332c-40e4-9317-9014c5749253\") " Nov 03 14:38:31.664636 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.664480 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8172c42-332c-40e4-9317-9014c5749253-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8172c42-332c-40e4-9317-9014c5749253-istio-envoy\") pod \"c8172c42-332c-40e4-9317-9014c5749253\" (UID: \"c8172c42-332c-40e4-9317-9014c5749253\") " Nov 03 14:38:31.664636 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.664511 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b671512d-59b5-4ddd-959d-1d9150d41268-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b671512d-59b5-4ddd-959d-1d9150d41268-workload-socket\") pod \"b671512d-59b5-4ddd-959d-1d9150d41268\" (UID: \"b671512d-59b5-4ddd-959d-1d9150d41268\") " Nov 03 14:38:31.664636 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.664556 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b671512d-59b5-4ddd-959d-1d9150d41268-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b671512d-59b5-4ddd-959d-1d9150d41268-credential-socket\") pod \"b671512d-59b5-4ddd-959d-1d9150d41268\" (UID: \"b671512d-59b5-4ddd-959d-1d9150d41268\") " Nov 03 14:38:31.664636 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.664613 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8172c42-332c-40e4-9317-9014c5749253-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8172c42-332c-40e4-9317-9014c5749253-istio-podinfo\") pod \"c8172c42-332c-40e4-9317-9014c5749253\" (UID: \"c8172c42-332c-40e4-9317-9014c5749253\") " Nov 03 14:38:31.664865 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.664652 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b671512d-59b5-4ddd-959d-1d9150d41268-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b671512d-59b5-4ddd-959d-1d9150d41268-workload-certs\") pod \"b671512d-59b5-4ddd-959d-1d9150d41268\" (UID: \"b671512d-59b5-4ddd-959d-1d9150d41268\") " Nov 03 14:38:31.664865 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.664680 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8172c42-332c-40e4-9317-9014c5749253-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8172c42-332c-40e4-9317-9014c5749253-istiod-ca-cert\") pod \"c8172c42-332c-40e4-9317-9014c5749253\" (UID: \"c8172c42-332c-40e4-9317-9014c5749253\") " Nov 03 14:38:31.664865 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.664767 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8172c42-332c-40e4-9317-9014c5749253-workload-socket" pod "c8172c42-332c-40e4-9317-9014c5749253" (UID: "c8172c42-332c-40e4-9317-9014c5749253"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:31.664865 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.664775 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b671512d-59b5-4ddd-959d-1d9150d41268-workload-socket" pod "b671512d-59b5-4ddd-959d-1d9150d41268" (UID: "b671512d-59b5-4ddd-959d-1d9150d41268"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:31.665077 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.665050 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b671512d-59b5-4ddd-959d-1d9150d41268-workload-certs" pod "b671512d-59b5-4ddd-959d-1d9150d41268" (UID: "b671512d-59b5-4ddd-959d-1d9150d41268"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:31.665142 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.665068 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b671512d-59b5-4ddd-959d-1d9150d41268-credential-socket" pod "b671512d-59b5-4ddd-959d-1d9150d41268" (UID: "b671512d-59b5-4ddd-959d-1d9150d41268"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:31.665142 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.665106 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b671512d-59b5-4ddd-959d-1d9150d41268-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b671512d-59b5-4ddd-959d-1d9150d41268-istio-envoy\") pod \"b671512d-59b5-4ddd-959d-1d9150d41268\" (UID: \"b671512d-59b5-4ddd-959d-1d9150d41268\") " Nov 03 14:38:31.665256 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.665148 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8172c42-332c-40e4-9317-9014c5749253-istiod-ca-cert" pod "c8172c42-332c-40e4-9317-9014c5749253" (UID: "c8172c42-332c-40e4-9317-9014c5749253"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:38:31.665256 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.665154 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8172c42-332c-40e4-9317-9014c5749253-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8172c42-332c-40e4-9317-9014c5749253-credential-socket\") pod \"c8172c42-332c-40e4-9317-9014c5749253\" (UID: \"c8172c42-332c-40e4-9317-9014c5749253\") " Nov 03 14:38:31.665256 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.665230 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8172c42-332c-40e4-9317-9014c5749253-istio-token\" (UniqueName: \"kubernetes.io/projected/c8172c42-332c-40e4-9317-9014c5749253-istio-token\") pod \"c8172c42-332c-40e4-9317-9014c5749253\" (UID: \"c8172c42-332c-40e4-9317-9014c5749253\") " Nov 03 14:38:31.665416 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.665271 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8172c42-332c-40e4-9317-9014c5749253-kube-api-access-dz2c9\" (UniqueName: \"kubernetes.io/projected/c8172c42-332c-40e4-9317-9014c5749253-kube-api-access-dz2c9\") pod \"c8172c42-332c-40e4-9317-9014c5749253\" (UID: \"c8172c42-332c-40e4-9317-9014c5749253\") " Nov 03 14:38:31.665416 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.665376 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8172c42-332c-40e4-9317-9014c5749253-credential-socket" pod "c8172c42-332c-40e4-9317-9014c5749253" (UID: "c8172c42-332c-40e4-9317-9014c5749253"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:31.665416 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.665407 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b671512d-59b5-4ddd-959d-1d9150d41268-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b671512d-59b5-4ddd-959d-1d9150d41268-istiod-ca-cert\") pod \"b671512d-59b5-4ddd-959d-1d9150d41268\" (UID: \"b671512d-59b5-4ddd-959d-1d9150d41268\") " Nov 03 14:38:31.665580 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.665474 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8172c42-332c-40e4-9317-9014c5749253-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8172c42-332c-40e4-9317-9014c5749253-workload-certs\") pod \"c8172c42-332c-40e4-9317-9014c5749253\" (UID: \"c8172c42-332c-40e4-9317-9014c5749253\") " Nov 03 14:38:31.665580 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.665512 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8172c42-332c-40e4-9317-9014c5749253-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8172c42-332c-40e4-9317-9014c5749253-istio-data\") pod \"c8172c42-332c-40e4-9317-9014c5749253\" (UID: \"c8172c42-332c-40e4-9317-9014c5749253\") " Nov 03 14:38:31.665580 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.665540 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b671512d-59b5-4ddd-959d-1d9150d41268-istio-token\" (UniqueName: \"kubernetes.io/projected/b671512d-59b5-4ddd-959d-1d9150d41268-istio-token\") pod \"b671512d-59b5-4ddd-959d-1d9150d41268\" (UID: \"b671512d-59b5-4ddd-959d-1d9150d41268\") " Nov 03 14:38:31.665745 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.665584 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b671512d-59b5-4ddd-959d-1d9150d41268-kube-api-access-vzlvn\" (UniqueName: \"kubernetes.io/projected/b671512d-59b5-4ddd-959d-1d9150d41268-kube-api-access-vzlvn\") pod \"b671512d-59b5-4ddd-959d-1d9150d41268\" (UID: \"b671512d-59b5-4ddd-959d-1d9150d41268\") " Nov 03 14:38:31.665745 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.665611 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b671512d-59b5-4ddd-959d-1d9150d41268-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b671512d-59b5-4ddd-959d-1d9150d41268-istio-data\") pod \"b671512d-59b5-4ddd-959d-1d9150d41268\" (UID: \"b671512d-59b5-4ddd-959d-1d9150d41268\") " Nov 03 14:38:31.665745 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.665644 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b671512d-59b5-4ddd-959d-1d9150d41268-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b671512d-59b5-4ddd-959d-1d9150d41268-istio-podinfo\") pod \"b671512d-59b5-4ddd-959d-1d9150d41268\" (UID: \"b671512d-59b5-4ddd-959d-1d9150d41268\") " Nov 03 14:38:31.666089 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.666063 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b671512d-59b5-4ddd-959d-1d9150d41268-istiod-ca-cert" pod "b671512d-59b5-4ddd-959d-1d9150d41268" (UID: "b671512d-59b5-4ddd-959d-1d9150d41268"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:38:31.666777 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.666221 5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f859590f-c6cb-4b6c-bfd2-ebde80002e81-istio-podinfo\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.666777 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.666240 5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b671512d-59b5-4ddd-959d-1d9150d41268-istiod-ca-cert\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.666777 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.666255 5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f859590f-c6cb-4b6c-bfd2-ebde80002e81-credential-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.666777 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.666269 5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f859590f-c6cb-4b6c-bfd2-ebde80002e81-workload-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.666777 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.666283 5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f859590f-c6cb-4b6c-bfd2-ebde80002e81-istio-token\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.666777 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.666297 5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8172c42-332c-40e4-9317-9014c5749253-workload-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.666777 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.666310 5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b671512d-59b5-4ddd-959d-1d9150d41268-workload-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.666777 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.666324 5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b671512d-59b5-4ddd-959d-1d9150d41268-credential-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.666777 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.666336 5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b671512d-59b5-4ddd-959d-1d9150d41268-workload-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.666777 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.666349 5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8172c42-332c-40e4-9317-9014c5749253-istiod-ca-cert\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.666777 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.666362 5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f859590f-c6cb-4b6c-bfd2-ebde80002e81-istio-envoy\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.666777 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.666375 5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f859590f-c6cb-4b6c-bfd2-ebde80002e81-workload-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.666777 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.666387 5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f859590f-c6cb-4b6c-bfd2-ebde80002e81-istio-data\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.666777 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.666401 5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f859590f-c6cb-4b6c-bfd2-ebde80002e81-istiod-ca-cert\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.666777 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.666414 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w2v7\" (UniqueName: \"kubernetes.io/projected/f859590f-c6cb-4b6c-bfd2-ebde80002e81-kube-api-access-8w2v7\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.666777 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.666427 5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8172c42-332c-40e4-9317-9014c5749253-credential-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.667684 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.667288 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8172c42-332c-40e4-9317-9014c5749253-istio-envoy" pod "c8172c42-332c-40e4-9317-9014c5749253" (UID: "c8172c42-332c-40e4-9317-9014c5749253"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:31.667684 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.667537 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8172c42-332c-40e4-9317-9014c5749253-istio-podinfo" pod "c8172c42-332c-40e4-9317-9014c5749253" (UID: "c8172c42-332c-40e4-9317-9014c5749253"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:38:31.667684 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.667583 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b671512d-59b5-4ddd-959d-1d9150d41268-istio-envoy" pod "b671512d-59b5-4ddd-959d-1d9150d41268" (UID: "b671512d-59b5-4ddd-959d-1d9150d41268"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:31.667861 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.667729 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8172c42-332c-40e4-9317-9014c5749253-istio-data" pod "c8172c42-332c-40e4-9317-9014c5749253" (UID: "c8172c42-332c-40e4-9317-9014c5749253"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:31.667861 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.667809 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8172c42-332c-40e4-9317-9014c5749253-workload-certs" pod "c8172c42-332c-40e4-9317-9014c5749253" (UID: "c8172c42-332c-40e4-9317-9014c5749253"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:31.667971 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.667823 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b671512d-59b5-4ddd-959d-1d9150d41268-istio-data" pod "b671512d-59b5-4ddd-959d-1d9150d41268" (UID: "b671512d-59b5-4ddd-959d-1d9150d41268"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:31.668804 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.668776 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b671512d-59b5-4ddd-959d-1d9150d41268-istio-podinfo" pod "b671512d-59b5-4ddd-959d-1d9150d41268" (UID: "b671512d-59b5-4ddd-959d-1d9150d41268"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:38:31.668962 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.668940 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8172c42-332c-40e4-9317-9014c5749253-kube-api-access-dz2c9" pod "c8172c42-332c-40e4-9317-9014c5749253" (UID: "c8172c42-332c-40e4-9317-9014c5749253"). InnerVolumeSpecName "kube-api-access-dz2c9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:31.669213 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.669189 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b671512d-59b5-4ddd-959d-1d9150d41268-istio-token" pod "b671512d-59b5-4ddd-959d-1d9150d41268" (UID: "b671512d-59b5-4ddd-959d-1d9150d41268"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:31.669307 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.669254 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b671512d-59b5-4ddd-959d-1d9150d41268-kube-api-access-vzlvn" pod "b671512d-59b5-4ddd-959d-1d9150d41268" (UID: "b671512d-59b5-4ddd-959d-1d9150d41268"). InnerVolumeSpecName "kube-api-access-vzlvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:31.669601 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.669588 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8172c42-332c-40e4-9317-9014c5749253-istio-token" pod "c8172c42-332c-40e4-9317-9014c5749253" (UID: "c8172c42-332c-40e4-9317-9014c5749253"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:31.767422 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.767390 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz2c9\" (UniqueName: \"kubernetes.io/projected/c8172c42-332c-40e4-9317-9014c5749253-kube-api-access-dz2c9\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.767422 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.767420 5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8172c42-332c-40e4-9317-9014c5749253-workload-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.767422 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.767430 5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8172c42-332c-40e4-9317-9014c5749253-istio-data\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.767660 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.767438 5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b671512d-59b5-4ddd-959d-1d9150d41268-istio-token\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.767660 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.767446 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzlvn\" (UniqueName: \"kubernetes.io/projected/b671512d-59b5-4ddd-959d-1d9150d41268-kube-api-access-vzlvn\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.767660 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.767455 5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b671512d-59b5-4ddd-959d-1d9150d41268-istio-data\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.767660 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.767464 5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b671512d-59b5-4ddd-959d-1d9150d41268-istio-podinfo\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.767660 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.767473 5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8172c42-332c-40e4-9317-9014c5749253-istio-envoy\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.767660 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.767484 5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8172c42-332c-40e4-9317-9014c5749253-istio-podinfo\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.767660 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.767492 5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b671512d-59b5-4ddd-959d-1d9150d41268-istio-envoy\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.767660 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.767500 5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8172c42-332c-40e4-9317-9014c5749253-istio-token\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:31.852723 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.852688 5060 generic.go:356] "Generic (PLEG): container finished" podID="f859590f-c6cb-4b6c-bfd2-ebde80002e81" containerID="34b172cbeb89aca4f1e6ca29cce8e5cfad22ead649e482baf802a82bb6b1e173" exitCode=0 Nov 03 14:38:31.852939 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.852753 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" Nov 03 14:38:31.852939 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.852773 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" event={"ID":"f859590f-c6cb-4b6c-bfd2-ebde80002e81","Type":"ContainerDied","Data":"34b172cbeb89aca4f1e6ca29cce8e5cfad22ead649e482baf802a82bb6b1e173"} Nov 03 14:38:31.852939 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.852879 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg" event={"ID":"f859590f-c6cb-4b6c-bfd2-ebde80002e81","Type":"ContainerDied","Data":"c6b3ecc656f10b4a4187b29872d4a55aa200453ac873adf84a4e80295c503f6f"} Nov 03 14:38:31.852939 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.852894 5060 scope.go:122] "RemoveContainer" containerID="34b172cbeb89aca4f1e6ca29cce8e5cfad22ead649e482baf802a82bb6b1e173" Nov 03 14:38:31.854667 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.854638 5060 generic.go:356] "Generic (PLEG): container finished" podID="b671512d-59b5-4ddd-959d-1d9150d41268" containerID="5e9e8520dd24462be6b1e66bcad82e3f8d92ffebe4a2569073dd63c91f586a01" exitCode=0 Nov 03 14:38:31.854774 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.854688 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" Nov 03 14:38:31.854774 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.854724 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" event={"ID":"b671512d-59b5-4ddd-959d-1d9150d41268","Type":"ContainerDied","Data":"5e9e8520dd24462be6b1e66bcad82e3f8d92ffebe4a2569073dd63c91f586a01"} Nov 03 14:38:31.854925 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.854774 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc" event={"ID":"b671512d-59b5-4ddd-959d-1d9150d41268","Type":"ContainerDied","Data":"1d31b71826d40bf0050ff127459e3b16e7f9958631a5727e0c88fbfbbf80fd20"} Nov 03 14:38:31.856495 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.856474 5060 generic.go:356] "Generic (PLEG): container finished" podID="c8172c42-332c-40e4-9317-9014c5749253" containerID="a1519cd1e4d0fa11fa27ab35ab2f35597611cc362f25997eae15530a155c8208" exitCode=0 Nov 03 14:38:31.856595 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.856537 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" Nov 03 14:38:31.856595 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.856555 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" event={"ID":"c8172c42-332c-40e4-9317-9014c5749253","Type":"ContainerDied","Data":"a1519cd1e4d0fa11fa27ab35ab2f35597611cc362f25997eae15530a155c8208"} Nov 03 14:38:31.856679 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.856619 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5" event={"ID":"c8172c42-332c-40e4-9317-9014c5749253","Type":"ContainerDied","Data":"eca94d3ad163e97b83e466654963add16b12f04a0c1128130243cc9c9bd83f78"} Nov 03 14:38:31.869059 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.869037 5060 scope.go:122] "RemoveContainer" containerID="34b172cbeb89aca4f1e6ca29cce8e5cfad22ead649e482baf802a82bb6b1e173" Nov 03 14:38:31.870120 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:38:31.870091 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34b172cbeb89aca4f1e6ca29cce8e5cfad22ead649e482baf802a82bb6b1e173\": container with ID starting with 34b172cbeb89aca4f1e6ca29cce8e5cfad22ead649e482baf802a82bb6b1e173 not found: ID does not exist" containerID="34b172cbeb89aca4f1e6ca29cce8e5cfad22ead649e482baf802a82bb6b1e173" Nov 03 14:38:31.870215 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.870128 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34b172cbeb89aca4f1e6ca29cce8e5cfad22ead649e482baf802a82bb6b1e173"} err="failed to get container status \"34b172cbeb89aca4f1e6ca29cce8e5cfad22ead649e482baf802a82bb6b1e173\": rpc error: code = NotFound desc = could not find container \"34b172cbeb89aca4f1e6ca29cce8e5cfad22ead649e482baf802a82bb6b1e173\": container with ID starting with 34b172cbeb89aca4f1e6ca29cce8e5cfad22ead649e482baf802a82bb6b1e173 not found: ID does not exist" Nov 03 14:38:31.870215 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.870151 5060 scope.go:122] "RemoveContainer" containerID="5e9e8520dd24462be6b1e66bcad82e3f8d92ffebe4a2569073dd63c91f586a01" Nov 03 14:38:31.879557 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.879542 5060 scope.go:122] "RemoveContainer" containerID="5e9e8520dd24462be6b1e66bcad82e3f8d92ffebe4a2569073dd63c91f586a01" Nov 03 14:38:31.879810 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:38:31.879794 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e9e8520dd24462be6b1e66bcad82e3f8d92ffebe4a2569073dd63c91f586a01\": container with ID starting with 5e9e8520dd24462be6b1e66bcad82e3f8d92ffebe4a2569073dd63c91f586a01 not found: ID does not exist" containerID="5e9e8520dd24462be6b1e66bcad82e3f8d92ffebe4a2569073dd63c91f586a01" Nov 03 14:38:31.879874 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.879816 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e9e8520dd24462be6b1e66bcad82e3f8d92ffebe4a2569073dd63c91f586a01"} err="failed to get container status \"5e9e8520dd24462be6b1e66bcad82e3f8d92ffebe4a2569073dd63c91f586a01\": rpc error: code = NotFound desc = could not find container \"5e9e8520dd24462be6b1e66bcad82e3f8d92ffebe4a2569073dd63c91f586a01\": container with ID starting with 5e9e8520dd24462be6b1e66bcad82e3f8d92ffebe4a2569073dd63c91f586a01 not found: ID does not exist" Nov 03 14:38:31.879874 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.879830 5060 scope.go:122] "RemoveContainer" containerID="a1519cd1e4d0fa11fa27ab35ab2f35597611cc362f25997eae15530a155c8208" Nov 03 14:38:31.889817 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.889790 5060 scope.go:122] "RemoveContainer" containerID="a1519cd1e4d0fa11fa27ab35ab2f35597611cc362f25997eae15530a155c8208" Nov 03 14:38:31.890096 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:38:31.890077 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1519cd1e4d0fa11fa27ab35ab2f35597611cc362f25997eae15530a155c8208\": container with ID starting with a1519cd1e4d0fa11fa27ab35ab2f35597611cc362f25997eae15530a155c8208 not found: ID does not exist" containerID="a1519cd1e4d0fa11fa27ab35ab2f35597611cc362f25997eae15530a155c8208" Nov 03 14:38:31.890148 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.890102 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1519cd1e4d0fa11fa27ab35ab2f35597611cc362f25997eae15530a155c8208"} err="failed to get container status \"a1519cd1e4d0fa11fa27ab35ab2f35597611cc362f25997eae15530a155c8208\": rpc error: code = NotFound desc = could not find container \"a1519cd1e4d0fa11fa27ab35ab2f35597611cc362f25997eae15530a155c8208\": container with ID starting with a1519cd1e4d0fa11fa27ab35ab2f35597611cc362f25997eae15530a155c8208 not found: ID does not exist" Nov 03 14:38:31.947615 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.947585 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg"] Nov 03 14:38:31.960413 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:31.960387 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-5shlg"] Nov 03 14:38:32.005745 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:32.005716 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc"] Nov 03 14:38:32.012059 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:32.012032 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-gqxxc"] Nov 03 14:38:32.048777 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:32.048702 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5"] Nov 03 14:38:32.055013 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:32.054988 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-z7fw5"] Nov 03 14:38:32.249023 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:32.248990 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b671512d-59b5-4ddd-959d-1d9150d41268" path="/var/lib/kubelet/pods/b671512d-59b5-4ddd-959d-1d9150d41268/volumes" Nov 03 14:38:32.249487 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:32.249474 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8172c42-332c-40e4-9317-9014c5749253" path="/var/lib/kubelet/pods/c8172c42-332c-40e4-9317-9014c5749253/volumes" Nov 03 14:38:32.249919 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:32.249905 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f859590f-c6cb-4b6c-bfd2-ebde80002e81" path="/var/lib/kubelet/pods/f859590f-c6cb-4b6c-bfd2-ebde80002e81/volumes" Nov 03 14:38:39.440667 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:38:39.440613 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:38:39.759731 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:39.759702 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4"] Nov 03 14:38:39.760209 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:39.760178 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" podUID="ab1a1c4a-c6c5-45bb-b3ee-529a355c3691" containerName="istio-proxy" containerID="cri-o://e8942975458ba43bb3ffab3ea5cde96288b5049eaa6cac0cb4797606df0c144c" gracePeriod=30 Nov 03 14:38:39.771092 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:39.771064 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf"] Nov 03 14:38:39.771367 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:39.771338 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" podUID="9ae45680-07cf-4414-851a-2272d68ccc21" containerName="istio-proxy" containerID="cri-o://6fc17d31664a5cac91b13b36795cac97370c2a816cf5cf94f81ac1fbafd2f6a2" gracePeriod=30 Nov 03 14:38:39.775877 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:39.775820 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9"] Nov 03 14:38:39.776082 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:39.776052 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" podUID="9a48c702-25f2-4601-9257-8281d6735a35" containerName="istio-proxy" containerID="cri-o://8b0afe4c6eefb518e1a65e06920e12673c31c0fa835cada1704cd43975dccb88" gracePeriod=30 Nov 03 14:38:39.781515 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:39.781491 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs"] Nov 03 14:38:39.781684 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:39.781666 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" podUID="ae832eba-b22b-41c2-8b30-a4e5d7d3afac" containerName="istio-proxy" containerID="cri-o://6850cc1b011df07b5b361cdb6d70008716799479a05dabfe2f459d76b92a8d85" gracePeriod=30 Nov 03 14:38:39.963990 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:39.963954 5060 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 14:38:39.964157 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:39.964007 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" podUID="ae832eba-b22b-41c2-8b30-a4e5d7d3afac" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 14:38:40.158920 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:40.158330 5060 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 14:38:40.158920 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:40.158389 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" podUID="9ae45680-07cf-4414-851a-2272d68ccc21" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 14:38:44.969129 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:44.968969 5060 generic.go:356] "Generic (PLEG): container finished" podID="9a48c702-25f2-4601-9257-8281d6735a35" containerID="8b0afe4c6eefb518e1a65e06920e12673c31c0fa835cada1704cd43975dccb88" exitCode=0 Nov 03 14:38:44.969129 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:44.969039 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" event={"ID":"9a48c702-25f2-4601-9257-8281d6735a35","Type":"ContainerDied","Data":"8b0afe4c6eefb518e1a65e06920e12673c31c0fa835cada1704cd43975dccb88"} Nov 03 14:38:44.980616 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:44.971656 5060 generic.go:356] "Generic (PLEG): container finished" podID="ab1a1c4a-c6c5-45bb-b3ee-529a355c3691" containerID="e8942975458ba43bb3ffab3ea5cde96288b5049eaa6cac0cb4797606df0c144c" exitCode=0 Nov 03 14:38:44.980616 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:44.971717 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" event={"ID":"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691","Type":"ContainerDied","Data":"e8942975458ba43bb3ffab3ea5cde96288b5049eaa6cac0cb4797606df0c144c"} Nov 03 14:38:44.980616 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:44.973630 5060 generic.go:356] "Generic (PLEG): container finished" podID="ae832eba-b22b-41c2-8b30-a4e5d7d3afac" containerID="6850cc1b011df07b5b361cdb6d70008716799479a05dabfe2f459d76b92a8d85" exitCode=0 Nov 03 14:38:44.980616 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:44.973741 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" event={"ID":"ae832eba-b22b-41c2-8b30-a4e5d7d3afac","Type":"ContainerDied","Data":"6850cc1b011df07b5b361cdb6d70008716799479a05dabfe2f459d76b92a8d85"} Nov 03 14:38:44.980616 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:44.976667 5060 generic.go:356] "Generic (PLEG): container finished" podID="9ae45680-07cf-4414-851a-2272d68ccc21" containerID="6fc17d31664a5cac91b13b36795cac97370c2a816cf5cf94f81ac1fbafd2f6a2" exitCode=0 Nov 03 14:38:44.980616 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:44.976711 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" event={"ID":"9ae45680-07cf-4414-851a-2272d68ccc21","Type":"ContainerDied","Data":"6fc17d31664a5cac91b13b36795cac97370c2a816cf5cf94f81ac1fbafd2f6a2"} Nov 03 14:38:45.063576 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.063248 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" Nov 03 14:38:45.134160 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.134066 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" Nov 03 14:38:45.178637 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.178609 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" Nov 03 14:38:45.195650 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.195624 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" Nov 03 14:38:45.236456 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.233390 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a48c702-25f2-4601-9257-8281d6735a35-istio-token\" (UniqueName: \"kubernetes.io/projected/9a48c702-25f2-4601-9257-8281d6735a35-istio-token\") pod \"9a48c702-25f2-4601-9257-8281d6735a35\" (UID: \"9a48c702-25f2-4601-9257-8281d6735a35\") " Nov 03 14:38:45.236456 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.233474 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-istiod-ca-cert\") pod \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\" (UID: \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\") " Nov 03 14:38:45.236456 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.233512 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a48c702-25f2-4601-9257-8281d6735a35-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a48c702-25f2-4601-9257-8281d6735a35-credential-socket\") pod \"9a48c702-25f2-4601-9257-8281d6735a35\" (UID: \"9a48c702-25f2-4601-9257-8281d6735a35\") " Nov 03 14:38:45.236456 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.233557 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a48c702-25f2-4601-9257-8281d6735a35-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a48c702-25f2-4601-9257-8281d6735a35-istio-podinfo\") pod \"9a48c702-25f2-4601-9257-8281d6735a35\" (UID: \"9a48c702-25f2-4601-9257-8281d6735a35\") " Nov 03 14:38:45.236456 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.233597 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-istio-envoy\") pod \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\" (UID: \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\") " Nov 03 14:38:45.236456 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.233667 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-credential-socket\") pod \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\" (UID: \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\") " Nov 03 14:38:45.236456 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.233709 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-workload-certs\") pod \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\" (UID: \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\") " Nov 03 14:38:45.236456 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.233748 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-istio-podinfo\") pod \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\" (UID: \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\") " Nov 03 14:38:45.236456 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.233774 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a48c702-25f2-4601-9257-8281d6735a35-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a48c702-25f2-4601-9257-8281d6735a35-istio-envoy\") pod \"9a48c702-25f2-4601-9257-8281d6735a35\" (UID: \"9a48c702-25f2-4601-9257-8281d6735a35\") " Nov 03 14:38:45.236456 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.233802 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a48c702-25f2-4601-9257-8281d6735a35-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a48c702-25f2-4601-9257-8281d6735a35-istiod-ca-cert\") pod \"9a48c702-25f2-4601-9257-8281d6735a35\" (UID: \"9a48c702-25f2-4601-9257-8281d6735a35\") " Nov 03 14:38:45.236456 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.233823 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a48c702-25f2-4601-9257-8281d6735a35-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a48c702-25f2-4601-9257-8281d6735a35-istio-data\") pod \"9a48c702-25f2-4601-9257-8281d6735a35\" (UID: \"9a48c702-25f2-4601-9257-8281d6735a35\") " Nov 03 14:38:45.236456 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.233890 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a48c702-25f2-4601-9257-8281d6735a35-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a48c702-25f2-4601-9257-8281d6735a35-workload-socket\") pod \"9a48c702-25f2-4601-9257-8281d6735a35\" (UID: \"9a48c702-25f2-4601-9257-8281d6735a35\") " Nov 03 14:38:45.236456 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.233921 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-workload-socket\") pod \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\" (UID: \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\") " Nov 03 14:38:45.236456 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.233948 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-istio-data\") pod \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\" (UID: \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\") " Nov 03 14:38:45.236456 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.233979 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a48c702-25f2-4601-9257-8281d6735a35-kube-api-access-vdg6g\" (UniqueName: \"kubernetes.io/projected/9a48c702-25f2-4601-9257-8281d6735a35-kube-api-access-vdg6g\") pod \"9a48c702-25f2-4601-9257-8281d6735a35\" (UID: \"9a48c702-25f2-4601-9257-8281d6735a35\") " Nov 03 14:38:45.236456 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.234044 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a48c702-25f2-4601-9257-8281d6735a35-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a48c702-25f2-4601-9257-8281d6735a35-workload-certs\") pod \"9a48c702-25f2-4601-9257-8281d6735a35\" (UID: \"9a48c702-25f2-4601-9257-8281d6735a35\") " Nov 03 14:38:45.236456 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.234098 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-istio-token\" (UniqueName: \"kubernetes.io/projected/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-istio-token\") pod \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\" (UID: \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\") " Nov 03 14:38:45.236456 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.234154 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-kube-api-access-xlcjp\" (UniqueName: \"kubernetes.io/projected/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-kube-api-access-xlcjp\") pod \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\" (UID: \"ae832eba-b22b-41c2-8b30-a4e5d7d3afac\") " Nov 03 14:38:45.236456 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.234178 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-credential-socket" pod "ae832eba-b22b-41c2-8b30-a4e5d7d3afac" (UID: "ae832eba-b22b-41c2-8b30-a4e5d7d3afac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:45.236456 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.235359 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a48c702-25f2-4601-9257-8281d6735a35-credential-socket" pod "9a48c702-25f2-4601-9257-8281d6735a35" (UID: "9a48c702-25f2-4601-9257-8281d6735a35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:45.236456 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.235461 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a48c702-25f2-4601-9257-8281d6735a35-istiod-ca-cert" pod "9a48c702-25f2-4601-9257-8281d6735a35" (UID: "9a48c702-25f2-4601-9257-8281d6735a35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:38:45.236456 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.235801 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-istiod-ca-cert" pod "ae832eba-b22b-41c2-8b30-a4e5d7d3afac" (UID: "ae832eba-b22b-41c2-8b30-a4e5d7d3afac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:38:45.236456 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.236155 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a48c702-25f2-4601-9257-8281d6735a35-workload-certs" pod "9a48c702-25f2-4601-9257-8281d6735a35" (UID: "9a48c702-25f2-4601-9257-8281d6735a35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:45.236456 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.236428 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a48c702-25f2-4601-9257-8281d6735a35-workload-socket" pod "9a48c702-25f2-4601-9257-8281d6735a35" (UID: "9a48c702-25f2-4601-9257-8281d6735a35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:45.245691 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.236618 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a48c702-25f2-4601-9257-8281d6735a35-istio-data" pod "9a48c702-25f2-4601-9257-8281d6735a35" (UID: "9a48c702-25f2-4601-9257-8281d6735a35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:45.245691 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.236825 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-workload-socket" pod "ae832eba-b22b-41c2-8b30-a4e5d7d3afac" (UID: "ae832eba-b22b-41c2-8b30-a4e5d7d3afac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:45.245691 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.237031 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-istio-data" pod "ae832eba-b22b-41c2-8b30-a4e5d7d3afac" (UID: "ae832eba-b22b-41c2-8b30-a4e5d7d3afac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:45.245691 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.237219 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-workload-certs" pod "ae832eba-b22b-41c2-8b30-a4e5d7d3afac" (UID: "ae832eba-b22b-41c2-8b30-a4e5d7d3afac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:45.245691 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.245230 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-istio-podinfo" pod "ae832eba-b22b-41c2-8b30-a4e5d7d3afac" (UID: "ae832eba-b22b-41c2-8b30-a4e5d7d3afac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:38:45.245691 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.245639 5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a48c702-25f2-4601-9257-8281d6735a35-istiod-ca-cert\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.245691 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.245659 5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a48c702-25f2-4601-9257-8281d6735a35-istio-data\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.245691 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.245673 5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a48c702-25f2-4601-9257-8281d6735a35-workload-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.245691 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.245689 5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-workload-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.246140 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.245705 5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-istio-data\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.246140 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.245719 5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a48c702-25f2-4601-9257-8281d6735a35-workload-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.246140 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.245732 5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-istiod-ca-cert\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.246140 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.245747 5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a48c702-25f2-4601-9257-8281d6735a35-credential-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.246140 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.245760 5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-credential-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.246140 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.245773 5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-workload-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.246140 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.245787 5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-istio-podinfo\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.247673 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.247640 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a48c702-25f2-4601-9257-8281d6735a35-kube-api-access-vdg6g" pod "9a48c702-25f2-4601-9257-8281d6735a35" (UID: "9a48c702-25f2-4601-9257-8281d6735a35"). InnerVolumeSpecName "kube-api-access-vdg6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:45.247855 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.247755 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a48c702-25f2-4601-9257-8281d6735a35-istio-podinfo" pod "9a48c702-25f2-4601-9257-8281d6735a35" (UID: "9a48c702-25f2-4601-9257-8281d6735a35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:38:45.250214 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.250192 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-istio-envoy" pod "ae832eba-b22b-41c2-8b30-a4e5d7d3afac" (UID: "ae832eba-b22b-41c2-8b30-a4e5d7d3afac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:45.250538 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.250490 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-istio-token" pod "ae832eba-b22b-41c2-8b30-a4e5d7d3afac" (UID: "ae832eba-b22b-41c2-8b30-a4e5d7d3afac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:45.251226 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.251200 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a48c702-25f2-4601-9257-8281d6735a35-istio-token" pod "9a48c702-25f2-4601-9257-8281d6735a35" (UID: "9a48c702-25f2-4601-9257-8281d6735a35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:45.251411 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.251393 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-kube-api-access-xlcjp" pod "ae832eba-b22b-41c2-8b30-a4e5d7d3afac" (UID: "ae832eba-b22b-41c2-8b30-a4e5d7d3afac"). InnerVolumeSpecName "kube-api-access-xlcjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:45.252400 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.252376 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a48c702-25f2-4601-9257-8281d6735a35-istio-envoy" pod "9a48c702-25f2-4601-9257-8281d6735a35" (UID: "9a48c702-25f2-4601-9257-8281d6735a35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:45.347037 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.346940 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ae45680-07cf-4414-851a-2272d68ccc21-istio-token\" (UniqueName: \"kubernetes.io/projected/9ae45680-07cf-4414-851a-2272d68ccc21-istio-token\") pod \"9ae45680-07cf-4414-851a-2272d68ccc21\" (UID: \"9ae45680-07cf-4414-851a-2272d68ccc21\") " Nov 03 14:38:45.347037 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.347022 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-kube-api-access-cwk5n\" (UniqueName: \"kubernetes.io/projected/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-kube-api-access-cwk5n\") pod \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\" (UID: \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\") " Nov 03 14:38:45.347266 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.347077 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-istio-token\" (UniqueName: \"kubernetes.io/projected/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-istio-token\") pod \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\" (UID: \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\") " Nov 03 14:38:45.347266 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.347154 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae45680-07cf-4414-851a-2272d68ccc21-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae45680-07cf-4414-851a-2272d68ccc21-workload-certs\") pod \"9ae45680-07cf-4414-851a-2272d68ccc21\" (UID: \"9ae45680-07cf-4414-851a-2272d68ccc21\") " Nov 03 14:38:45.347266 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.347186 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-istio-podinfo\") pod \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\" (UID: \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\") " Nov 03 14:38:45.347266 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.347238 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-istio-envoy\") pod \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\" (UID: \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\") " Nov 03 14:38:45.347464 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.347278 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ae45680-07cf-4414-851a-2272d68ccc21-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae45680-07cf-4414-851a-2272d68ccc21-istio-podinfo\") pod \"9ae45680-07cf-4414-851a-2272d68ccc21\" (UID: \"9ae45680-07cf-4414-851a-2272d68ccc21\") " Nov 03 14:38:45.347464 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.347306 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae45680-07cf-4414-851a-2272d68ccc21-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae45680-07cf-4414-851a-2272d68ccc21-credential-socket\") pod \"9ae45680-07cf-4414-851a-2272d68ccc21\" (UID: \"9ae45680-07cf-4414-851a-2272d68ccc21\") " Nov 03 14:38:45.347464 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.347334 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ae45680-07cf-4414-851a-2272d68ccc21-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae45680-07cf-4414-851a-2272d68ccc21-istiod-ca-cert\") pod \"9ae45680-07cf-4414-851a-2272d68ccc21\" (UID: \"9ae45680-07cf-4414-851a-2272d68ccc21\") " Nov 03 14:38:45.347464 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.347366 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-istio-data\") pod \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\" (UID: \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\") " Nov 03 14:38:45.347464 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.347392 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae45680-07cf-4414-851a-2272d68ccc21-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae45680-07cf-4414-851a-2272d68ccc21-istio-envoy\") pod \"9ae45680-07cf-4414-851a-2272d68ccc21\" (UID: \"9ae45680-07cf-4414-851a-2272d68ccc21\") " Nov 03 14:38:45.347689 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.347464 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae45680-07cf-4414-851a-2272d68ccc21-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae45680-07cf-4414-851a-2272d68ccc21-istio-data\") pod \"9ae45680-07cf-4414-851a-2272d68ccc21\" (UID: \"9ae45680-07cf-4414-851a-2272d68ccc21\") " Nov 03 14:38:45.347689 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.347503 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-workload-socket\") pod \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\" (UID: \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\") " Nov 03 14:38:45.347689 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.347513 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae45680-07cf-4414-851a-2272d68ccc21-workload-certs" pod "9ae45680-07cf-4414-851a-2272d68ccc21" (UID: "9ae45680-07cf-4414-851a-2272d68ccc21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:45.347689 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.347536 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-credential-socket\") pod \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\" (UID: \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\") " Nov 03 14:38:45.347689 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.347565 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-workload-certs\") pod \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\" (UID: \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\") " Nov 03 14:38:45.347689 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.347603 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae45680-07cf-4414-851a-2272d68ccc21-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae45680-07cf-4414-851a-2272d68ccc21-workload-socket\") pod \"9ae45680-07cf-4414-851a-2272d68ccc21\" (UID: \"9ae45680-07cf-4414-851a-2272d68ccc21\") " Nov 03 14:38:45.348098 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.347721 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-istiod-ca-cert\") pod \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\" (UID: \"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691\") " Nov 03 14:38:45.348098 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.347765 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ae45680-07cf-4414-851a-2272d68ccc21-kube-api-access-7d4ll\" (UniqueName: \"kubernetes.io/projected/9ae45680-07cf-4414-851a-2272d68ccc21-kube-api-access-7d4ll\") pod \"9ae45680-07cf-4414-851a-2272d68ccc21\" (UID: \"9ae45680-07cf-4414-851a-2272d68ccc21\") " Nov 03 14:38:45.348098 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.347924 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ae45680-07cf-4414-851a-2272d68ccc21-istiod-ca-cert" pod "9ae45680-07cf-4414-851a-2272d68ccc21" (UID: "9ae45680-07cf-4414-851a-2272d68ccc21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:38:45.348330 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.348306 5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-istio-envoy\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.348397 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.348337 5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a48c702-25f2-4601-9257-8281d6735a35-istio-envoy\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.348397 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.348355 5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae45680-07cf-4414-851a-2272d68ccc21-workload-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.348397 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.348371 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdg6g\" (UniqueName: \"kubernetes.io/projected/9a48c702-25f2-4601-9257-8281d6735a35-kube-api-access-vdg6g\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.348397 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.348388 5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae45680-07cf-4414-851a-2272d68ccc21-istiod-ca-cert\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.348604 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.348402 5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-istio-token\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.348604 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.348417 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlcjp\" (UniqueName: \"kubernetes.io/projected/ae832eba-b22b-41c2-8b30-a4e5d7d3afac-kube-api-access-xlcjp\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.348604 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.348431 5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a48c702-25f2-4601-9257-8281d6735a35-istio-token\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.348604 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.348446 5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a48c702-25f2-4601-9257-8281d6735a35-istio-podinfo\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.348808 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.348787 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae45680-07cf-4414-851a-2272d68ccc21-credential-socket" pod "9ae45680-07cf-4414-851a-2272d68ccc21" (UID: "9ae45680-07cf-4414-851a-2272d68ccc21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:45.349025 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.348995 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-workload-socket" pod "ab1a1c4a-c6c5-45bb-b3ee-529a355c3691" (UID: "ab1a1c4a-c6c5-45bb-b3ee-529a355c3691"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:45.349207 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.349185 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-istio-data" pod "ab1a1c4a-c6c5-45bb-b3ee-529a355c3691" (UID: "ab1a1c4a-c6c5-45bb-b3ee-529a355c3691"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:45.350016 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.349984 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ae45680-07cf-4414-851a-2272d68ccc21-istio-token" pod "9ae45680-07cf-4414-851a-2272d68ccc21" (UID: "9ae45680-07cf-4414-851a-2272d68ccc21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:45.350016 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.349994 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-istio-envoy" pod "ab1a1c4a-c6c5-45bb-b3ee-529a355c3691" (UID: "ab1a1c4a-c6c5-45bb-b3ee-529a355c3691"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:45.350200 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.350023 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-istio-token" pod "ab1a1c4a-c6c5-45bb-b3ee-529a355c3691" (UID: "ab1a1c4a-c6c5-45bb-b3ee-529a355c3691"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:45.350200 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.350065 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-istio-podinfo" pod "ab1a1c4a-c6c5-45bb-b3ee-529a355c3691" (UID: "ab1a1c4a-c6c5-45bb-b3ee-529a355c3691"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:38:45.350200 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.350163 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-workload-certs" pod "ab1a1c4a-c6c5-45bb-b3ee-529a355c3691" (UID: "ab1a1c4a-c6c5-45bb-b3ee-529a355c3691"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:45.350200 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.350182 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ae45680-07cf-4414-851a-2272d68ccc21-kube-api-access-7d4ll" pod "9ae45680-07cf-4414-851a-2272d68ccc21" (UID: "9ae45680-07cf-4414-851a-2272d68ccc21"). InnerVolumeSpecName "kube-api-access-7d4ll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:45.350443 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.350282 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae45680-07cf-4414-851a-2272d68ccc21-istio-data" pod "9ae45680-07cf-4414-851a-2272d68ccc21" (UID: "9ae45680-07cf-4414-851a-2272d68ccc21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:45.350443 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.350295 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae45680-07cf-4414-851a-2272d68ccc21-workload-socket" pod "9ae45680-07cf-4414-851a-2272d68ccc21" (UID: "9ae45680-07cf-4414-851a-2272d68ccc21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:45.350443 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.350365 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-credential-socket" pod "ab1a1c4a-c6c5-45bb-b3ee-529a355c3691" (UID: "ab1a1c4a-c6c5-45bb-b3ee-529a355c3691"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:45.350443 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.350404 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-istiod-ca-cert" pod "ab1a1c4a-c6c5-45bb-b3ee-529a355c3691" (UID: "ab1a1c4a-c6c5-45bb-b3ee-529a355c3691"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:38:45.351374 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.351352 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae45680-07cf-4414-851a-2272d68ccc21-istio-envoy" pod "9ae45680-07cf-4414-851a-2272d68ccc21" (UID: "9ae45680-07cf-4414-851a-2272d68ccc21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:38:45.351567 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.351540 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-kube-api-access-cwk5n" pod "ab1a1c4a-c6c5-45bb-b3ee-529a355c3691" (UID: "ab1a1c4a-c6c5-45bb-b3ee-529a355c3691"). InnerVolumeSpecName "kube-api-access-cwk5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:45.351805 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.351787 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ae45680-07cf-4414-851a-2272d68ccc21-istio-podinfo" pod "9ae45680-07cf-4414-851a-2272d68ccc21" (UID: "9ae45680-07cf-4414-851a-2272d68ccc21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:38:45.449618 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.449588 5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-istio-data\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.449618 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.449614 5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae45680-07cf-4414-851a-2272d68ccc21-istio-envoy\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.449618 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.449622 5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae45680-07cf-4414-851a-2272d68ccc21-istio-data\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.449872 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.449631 5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-workload-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.449872 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.449640 5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-credential-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.449872 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.449648 5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-workload-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.449872 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.449659 5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae45680-07cf-4414-851a-2272d68ccc21-workload-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.449872 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.449668 5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-istiod-ca-cert\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.449872 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.449677 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d4ll\" (UniqueName: \"kubernetes.io/projected/9ae45680-07cf-4414-851a-2272d68ccc21-kube-api-access-7d4ll\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.449872 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.449686 5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ae45680-07cf-4414-851a-2272d68ccc21-istio-token\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.449872 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.449694 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwk5n\" (UniqueName: \"kubernetes.io/projected/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-kube-api-access-cwk5n\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.449872 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.449702 5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-istio-token\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.449872 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.449710 5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-istio-podinfo\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.449872 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.449717 5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691-istio-envoy\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.449872 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.449726 5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae45680-07cf-4414-851a-2272d68ccc21-istio-podinfo\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.449872 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.449735 5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae45680-07cf-4414-851a-2272d68ccc21-credential-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:38:45.984914 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.984882 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" Nov 03 14:38:45.984914 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.984902 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9" event={"ID":"9a48c702-25f2-4601-9257-8281d6735a35","Type":"ContainerDied","Data":"6a83c679ab82c23a68f5b135cabb88c0edbae3ea1189b5692163f548d28fe70a"} Nov 03 14:38:45.985470 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.984973 5060 scope.go:122] "RemoveContainer" containerID="8b0afe4c6eefb518e1a65e06920e12673c31c0fa835cada1704cd43975dccb88" Nov 03 14:38:45.986585 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.986563 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" Nov 03 14:38:45.986585 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.986570 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4" event={"ID":"ab1a1c4a-c6c5-45bb-b3ee-529a355c3691","Type":"ContainerDied","Data":"11f34292c1b8574acba8f59c4492146bcc9245b7efbed2f67e9c6412bcfcb409"} Nov 03 14:38:45.988310 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.988173 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" Nov 03 14:38:45.988310 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.988218 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs" event={"ID":"ae832eba-b22b-41c2-8b30-a4e5d7d3afac","Type":"ContainerDied","Data":"9109e1d75e0db74e060ef4c990b507ca43624139c63c7c20fbe95bfa21c137f8"} Nov 03 14:38:45.990870 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.989927 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" Nov 03 14:38:45.990870 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.989930 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf" event={"ID":"9ae45680-07cf-4414-851a-2272d68ccc21","Type":"ContainerDied","Data":"0bdaafa2927aeadec2e9468abe7d9a5e2b7c35a4eccd9a5db093c730c04234bf"} Nov 03 14:38:45.999014 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:45.998989 5060 scope.go:122] "RemoveContainer" containerID="e8942975458ba43bb3ffab3ea5cde96288b5049eaa6cac0cb4797606df0c144c" Nov 03 14:38:46.008920 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:46.008901 5060 scope.go:122] "RemoveContainer" containerID="6850cc1b011df07b5b361cdb6d70008716799479a05dabfe2f459d76b92a8d85" Nov 03 14:38:46.017391 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:46.017375 5060 scope.go:122] "RemoveContainer" containerID="6fc17d31664a5cac91b13b36795cac97370c2a816cf5cf94f81ac1fbafd2f6a2" Nov 03 14:38:46.058236 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:46.058176 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf"] Nov 03 14:38:46.070608 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:46.070582 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-765kf"] Nov 03 14:38:46.105794 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:46.105757 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs"] Nov 03 14:38:46.118441 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:46.118409 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lnlrs"] Nov 03 14:38:46.157864 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:46.157822 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9"] Nov 03 14:38:46.166584 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:46.166558 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-64pn9"] Nov 03 14:38:46.203669 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:46.203639 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4"] Nov 03 14:38:46.216379 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:46.216351 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-lcsl4"] Nov 03 14:38:46.258324 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:46.258290 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a48c702-25f2-4601-9257-8281d6735a35" path="/var/lib/kubelet/pods/9a48c702-25f2-4601-9257-8281d6735a35/volumes" Nov 03 14:38:46.258956 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:46.258935 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ae45680-07cf-4414-851a-2272d68ccc21" path="/var/lib/kubelet/pods/9ae45680-07cf-4414-851a-2272d68ccc21/volumes" Nov 03 14:38:46.259527 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:46.259510 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab1a1c4a-c6c5-45bb-b3ee-529a355c3691" path="/var/lib/kubelet/pods/ab1a1c4a-c6c5-45bb-b3ee-529a355c3691/volumes" Nov 03 14:38:46.260564 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:38:46.260545 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae832eba-b22b-41c2-8b30-a4e5d7d3afac" path="/var/lib/kubelet/pods/ae832eba-b22b-41c2-8b30-a4e5d7d3afac/volumes" Nov 03 14:39:04.266543 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:04.266512 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz"] Nov 03 14:39:04.267147 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:04.266707 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" podUID="50995ad2-461f-44c5-97e9-8e4f1922e39d" containerName="istio-proxy" containerID="cri-o://34cc929db15530dee75908267588804436663ee93d42c1d191f292ae9a1705cc" gracePeriod=30 Nov 03 14:39:09.444458 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:39:09.444414 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:39:09.521292 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:09.521260 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" Nov 03 14:39:09.589900 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:09.589794 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50995ad2-461f-44c5-97e9-8e4f1922e39d-istio-token\" (UniqueName: \"kubernetes.io/projected/50995ad2-461f-44c5-97e9-8e4f1922e39d-istio-token\") pod \"50995ad2-461f-44c5-97e9-8e4f1922e39d\" (UID: \"50995ad2-461f-44c5-97e9-8e4f1922e39d\") " Nov 03 14:39:09.589900 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:09.589890 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50995ad2-461f-44c5-97e9-8e4f1922e39d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50995ad2-461f-44c5-97e9-8e4f1922e39d-workload-certs\") pod \"50995ad2-461f-44c5-97e9-8e4f1922e39d\" (UID: \"50995ad2-461f-44c5-97e9-8e4f1922e39d\") " Nov 03 14:39:09.590109 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:09.589982 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50995ad2-461f-44c5-97e9-8e4f1922e39d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50995ad2-461f-44c5-97e9-8e4f1922e39d-istiod-ca-cert\") pod \"50995ad2-461f-44c5-97e9-8e4f1922e39d\" (UID: \"50995ad2-461f-44c5-97e9-8e4f1922e39d\") " Nov 03 14:39:09.590109 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:09.590073 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50995ad2-461f-44c5-97e9-8e4f1922e39d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50995ad2-461f-44c5-97e9-8e4f1922e39d-istio-data\") pod \"50995ad2-461f-44c5-97e9-8e4f1922e39d\" (UID: \"50995ad2-461f-44c5-97e9-8e4f1922e39d\") " Nov 03 14:39:09.590234 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:09.590121 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50995ad2-461f-44c5-97e9-8e4f1922e39d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50995ad2-461f-44c5-97e9-8e4f1922e39d-istio-envoy\") pod \"50995ad2-461f-44c5-97e9-8e4f1922e39d\" (UID: \"50995ad2-461f-44c5-97e9-8e4f1922e39d\") " Nov 03 14:39:09.590234 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:09.590183 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50995ad2-461f-44c5-97e9-8e4f1922e39d-kube-api-access-rznzp\" (UniqueName: \"kubernetes.io/projected/50995ad2-461f-44c5-97e9-8e4f1922e39d-kube-api-access-rznzp\") pod \"50995ad2-461f-44c5-97e9-8e4f1922e39d\" (UID: \"50995ad2-461f-44c5-97e9-8e4f1922e39d\") " Nov 03 14:39:09.590234 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:09.590227 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50995ad2-461f-44c5-97e9-8e4f1922e39d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50995ad2-461f-44c5-97e9-8e4f1922e39d-istio-podinfo\") pod \"50995ad2-461f-44c5-97e9-8e4f1922e39d\" (UID: \"50995ad2-461f-44c5-97e9-8e4f1922e39d\") " Nov 03 14:39:09.590410 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:09.590241 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50995ad2-461f-44c5-97e9-8e4f1922e39d-workload-certs" pod "50995ad2-461f-44c5-97e9-8e4f1922e39d" (UID: "50995ad2-461f-44c5-97e9-8e4f1922e39d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:39:09.590410 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:09.590268 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50995ad2-461f-44c5-97e9-8e4f1922e39d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50995ad2-461f-44c5-97e9-8e4f1922e39d-workload-socket\") pod \"50995ad2-461f-44c5-97e9-8e4f1922e39d\" (UID: \"50995ad2-461f-44c5-97e9-8e4f1922e39d\") " Nov 03 14:39:09.590410 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:09.590339 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50995ad2-461f-44c5-97e9-8e4f1922e39d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50995ad2-461f-44c5-97e9-8e4f1922e39d-credential-socket\") pod \"50995ad2-461f-44c5-97e9-8e4f1922e39d\" (UID: \"50995ad2-461f-44c5-97e9-8e4f1922e39d\") " Nov 03 14:39:09.590410 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:09.590353 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50995ad2-461f-44c5-97e9-8e4f1922e39d-istio-data" pod "50995ad2-461f-44c5-97e9-8e4f1922e39d" (UID: "50995ad2-461f-44c5-97e9-8e4f1922e39d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:39:09.590678 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:09.590427 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50995ad2-461f-44c5-97e9-8e4f1922e39d-istiod-ca-cert" pod "50995ad2-461f-44c5-97e9-8e4f1922e39d" (UID: "50995ad2-461f-44c5-97e9-8e4f1922e39d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:39:09.591044 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:09.591017 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50995ad2-461f-44c5-97e9-8e4f1922e39d-workload-socket" pod "50995ad2-461f-44c5-97e9-8e4f1922e39d" (UID: "50995ad2-461f-44c5-97e9-8e4f1922e39d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:39:09.591163 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:09.591022 5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50995ad2-461f-44c5-97e9-8e4f1922e39d-workload-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:39:09.591163 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:09.591074 5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50995ad2-461f-44c5-97e9-8e4f1922e39d-istiod-ca-cert\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:39:09.591163 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:09.591077 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50995ad2-461f-44c5-97e9-8e4f1922e39d-credential-socket" pod "50995ad2-461f-44c5-97e9-8e4f1922e39d" (UID: "50995ad2-461f-44c5-97e9-8e4f1922e39d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:39:09.591163 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:09.591094 5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50995ad2-461f-44c5-97e9-8e4f1922e39d-istio-data\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:39:09.592693 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:09.592667 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50995ad2-461f-44c5-97e9-8e4f1922e39d-istio-envoy" pod "50995ad2-461f-44c5-97e9-8e4f1922e39d" (UID: "50995ad2-461f-44c5-97e9-8e4f1922e39d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:39:09.592957 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:09.592934 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50995ad2-461f-44c5-97e9-8e4f1922e39d-istio-token" pod "50995ad2-461f-44c5-97e9-8e4f1922e39d" (UID: "50995ad2-461f-44c5-97e9-8e4f1922e39d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:39:09.593057 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:09.592954 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50995ad2-461f-44c5-97e9-8e4f1922e39d-kube-api-access-rznzp" pod "50995ad2-461f-44c5-97e9-8e4f1922e39d" (UID: "50995ad2-461f-44c5-97e9-8e4f1922e39d"). InnerVolumeSpecName "kube-api-access-rznzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:39:09.593057 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:09.592962 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50995ad2-461f-44c5-97e9-8e4f1922e39d-istio-podinfo" pod "50995ad2-461f-44c5-97e9-8e4f1922e39d" (UID: "50995ad2-461f-44c5-97e9-8e4f1922e39d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:39:09.692562 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:09.692529 5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50995ad2-461f-44c5-97e9-8e4f1922e39d-istio-envoy\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:39:09.692562 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:09.692558 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rznzp\" (UniqueName: \"kubernetes.io/projected/50995ad2-461f-44c5-97e9-8e4f1922e39d-kube-api-access-rznzp\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:39:09.692562 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:09.692569 5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50995ad2-461f-44c5-97e9-8e4f1922e39d-istio-podinfo\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:39:09.692793 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:09.692578 5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50995ad2-461f-44c5-97e9-8e4f1922e39d-workload-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:39:09.692793 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:09.692587 5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50995ad2-461f-44c5-97e9-8e4f1922e39d-credential-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:39:09.692793 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:09.692596 5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50995ad2-461f-44c5-97e9-8e4f1922e39d-istio-token\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:39:10.251083 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:10.251048 5060 generic.go:356] "Generic (PLEG): container finished" podID="50995ad2-461f-44c5-97e9-8e4f1922e39d" containerID="34cc929db15530dee75908267588804436663ee93d42c1d191f292ae9a1705cc" exitCode=0 Nov 03 14:39:10.251277 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:10.251110 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" Nov 03 14:39:10.251277 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:10.251130 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" event={"ID":"50995ad2-461f-44c5-97e9-8e4f1922e39d","Type":"ContainerDied","Data":"34cc929db15530dee75908267588804436663ee93d42c1d191f292ae9a1705cc"} Nov 03 14:39:10.251277 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:10.251187 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz" event={"ID":"50995ad2-461f-44c5-97e9-8e4f1922e39d","Type":"ContainerDied","Data":"48db54768d0e4a64fcf3f677ab6276dae1d21b57839c7e20e96ee4abdcdbb2eb"} Nov 03 14:39:10.251277 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:10.251219 5060 scope.go:122] "RemoveContainer" containerID="34cc929db15530dee75908267588804436663ee93d42c1d191f292ae9a1705cc" Nov 03 14:39:10.261029 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:10.261008 5060 scope.go:122] "RemoveContainer" containerID="34cc929db15530dee75908267588804436663ee93d42c1d191f292ae9a1705cc" Nov 03 14:39:10.261314 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:39:10.261294 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34cc929db15530dee75908267588804436663ee93d42c1d191f292ae9a1705cc\": container with ID starting with 34cc929db15530dee75908267588804436663ee93d42c1d191f292ae9a1705cc not found: ID does not exist" containerID="34cc929db15530dee75908267588804436663ee93d42c1d191f292ae9a1705cc" Nov 03 14:39:10.261376 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:10.261324 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34cc929db15530dee75908267588804436663ee93d42c1d191f292ae9a1705cc"} err="failed to get container status \"34cc929db15530dee75908267588804436663ee93d42c1d191f292ae9a1705cc\": rpc error: code = NotFound desc = could not find container \"34cc929db15530dee75908267588804436663ee93d42c1d191f292ae9a1705cc\": container with ID starting with 34cc929db15530dee75908267588804436663ee93d42c1d191f292ae9a1705cc not found: ID does not exist" Nov 03 14:39:10.322956 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:10.322927 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz"] Nov 03 14:39:10.335989 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:10.335962 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-tdlcz"] Nov 03 14:39:12.248400 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:12.248367 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50995ad2-461f-44c5-97e9-8e4f1922e39d" path="/var/lib/kubelet/pods/50995ad2-461f-44c5-97e9-8e4f1922e39d/volumes" Nov 03 14:39:18.587859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:18.586944 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:39:18.587859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:18.587035 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" podUID="43465f0f-764a-44c5-b762-9b6932953b41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:39:19.279976 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:19.279937 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-rr46x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:39:19.287862 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:19.280242 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" podUID="50a3ab6a-5b07-4d38-b00b-9b77b506ec20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:39:19.372566 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:19.372000 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:39:19.372566 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:19.372072 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" podUID="2d7d5e9c-625a-42af-98d7-52b9a8b5745e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:39:24.697939 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.697902 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf4400e7-7bpsm"] Nov 03 14:39:24.698379 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.698364 5060 cpu_manager.go:414] "containerMap: removing container" podUID="6c8d7e93-3520-4a18-916a-ac77ae09f885" containerName="istio-proxy" Nov 03 14:39:24.698427 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.698381 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c8d7e93-3520-4a18-916a-ac77ae09f885" containerName="istio-proxy" Nov 03 14:39:24.698427 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.698395 5060 cpu_manager.go:414] "containerMap: removing container" podUID="9ae45680-07cf-4414-851a-2272d68ccc21" containerName="istio-proxy" Nov 03 14:39:24.698427 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.698401 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ae45680-07cf-4414-851a-2272d68ccc21" containerName="istio-proxy" Nov 03 14:39:24.698427 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.698419 5060 cpu_manager.go:414] "containerMap: removing container" podUID="ab1a1c4a-c6c5-45bb-b3ee-529a355c3691" containerName="istio-proxy" Nov 03 14:39:24.698427 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.698425 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab1a1c4a-c6c5-45bb-b3ee-529a355c3691" containerName="istio-proxy" Nov 03 14:39:24.698580 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.698436 5060 cpu_manager.go:414] "containerMap: removing container" podUID="2aafae9b-8aa6-46fe-bb81-a37c0e52fff7" containerName="istio-proxy" Nov 03 14:39:24.698580 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.698441 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aafae9b-8aa6-46fe-bb81-a37c0e52fff7" containerName="istio-proxy" Nov 03 14:39:24.698580 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.698451 5060 cpu_manager.go:414] "containerMap: removing container" podUID="50995ad2-461f-44c5-97e9-8e4f1922e39d" containerName="istio-proxy" Nov 03 14:39:24.698580 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.698455 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50995ad2-461f-44c5-97e9-8e4f1922e39d" containerName="istio-proxy" Nov 03 14:39:24.698580 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.698462 5060 cpu_manager.go:414] "containerMap: removing container" podUID="9a48c702-25f2-4601-9257-8281d6735a35" containerName="istio-proxy" Nov 03 14:39:24.698580 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.698466 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a48c702-25f2-4601-9257-8281d6735a35" containerName="istio-proxy" Nov 03 14:39:24.698580 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.698474 5060 cpu_manager.go:414] "containerMap: removing container" podUID="f859590f-c6cb-4b6c-bfd2-ebde80002e81" containerName="istio-proxy" Nov 03 14:39:24.698580 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.698478 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f859590f-c6cb-4b6c-bfd2-ebde80002e81" containerName="istio-proxy" Nov 03 14:39:24.698580 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.698486 5060 cpu_manager.go:414] "containerMap: removing container" podUID="6c8d7e93-3520-4a18-916a-ac77ae09f885" containerName="istio-proxy" Nov 03 14:39:24.698580 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.698491 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c8d7e93-3520-4a18-916a-ac77ae09f885" containerName="istio-proxy" Nov 03 14:39:24.698580 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.698497 5060 cpu_manager.go:414] "containerMap: removing container" podUID="b671512d-59b5-4ddd-959d-1d9150d41268" containerName="istio-proxy" Nov 03 14:39:24.698580 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.698501 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b671512d-59b5-4ddd-959d-1d9150d41268" containerName="istio-proxy" Nov 03 14:39:24.698580 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.698507 5060 cpu_manager.go:414] "containerMap: removing container" podUID="c8172c42-332c-40e4-9317-9014c5749253" containerName="istio-proxy" Nov 03 14:39:24.698580 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.698511 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8172c42-332c-40e4-9317-9014c5749253" containerName="istio-proxy" Nov 03 14:39:24.698580 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.698516 5060 cpu_manager.go:414] "containerMap: removing container" podUID="0cb8e7eb-3150-46e8-b9b6-48f53b265483" containerName="revoke-keys" Nov 03 14:39:24.698580 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.698520 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cb8e7eb-3150-46e8-b9b6-48f53b265483" containerName="revoke-keys" Nov 03 14:39:24.698580 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.698532 5060 cpu_manager.go:414] "containerMap: removing container" podUID="ae832eba-b22b-41c2-8b30-a4e5d7d3afac" containerName="istio-proxy" Nov 03 14:39:24.698580 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.698536 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae832eba-b22b-41c2-8b30-a4e5d7d3afac" containerName="istio-proxy" Nov 03 14:39:24.699125 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.698613 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c8d7e93-3520-4a18-916a-ac77ae09f885" containerName="istio-proxy" Nov 03 14:39:24.699125 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.698621 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ae45680-07cf-4414-851a-2272d68ccc21" containerName="istio-proxy" Nov 03 14:39:24.699125 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.698629 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab1a1c4a-c6c5-45bb-b3ee-529a355c3691" containerName="istio-proxy" Nov 03 14:39:24.699125 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.698637 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aafae9b-8aa6-46fe-bb81-a37c0e52fff7" containerName="istio-proxy" Nov 03 14:39:24.699125 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.698642 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="50995ad2-461f-44c5-97e9-8e4f1922e39d" containerName="istio-proxy" Nov 03 14:39:24.699125 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.698647 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a48c702-25f2-4601-9257-8281d6735a35" containerName="istio-proxy" Nov 03 14:39:24.699125 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.698653 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f859590f-c6cb-4b6c-bfd2-ebde80002e81" containerName="istio-proxy" Nov 03 14:39:24.699125 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.698659 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="b671512d-59b5-4ddd-959d-1d9150d41268" containerName="istio-proxy" Nov 03 14:39:24.699125 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.698664 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8172c42-332c-40e4-9317-9014c5749253" containerName="istio-proxy" Nov 03 14:39:24.699125 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.698668 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cb8e7eb-3150-46e8-b9b6-48f53b265483" containerName="revoke-keys" Nov 03 14:39:24.699125 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.698674 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae832eba-b22b-41c2-8b30-a4e5d7d3afac" containerName="istio-proxy" Nov 03 14:39:24.703057 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.703036 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4400e7-7bpsm" Nov 03 14:39:24.727533 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.727504 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf4400e7-7bpsm"] Nov 03 14:39:24.736736 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.736709 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:39:24.736953 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.736761 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:39:24.736953 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.736863 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\"" Nov 03 14:39:24.737094 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.736918 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:39:24.776564 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.776526 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xqs4\" (UniqueName: \"kubernetes.io/projected/e634e8b3-2f74-482d-aaf5-4165f8294b45-kube-api-access-8xqs4\") pod \"maas-api-revoke-sub-cf4400e7-7bpsm\" (UID: \"e634e8b3-2f74-482d-aaf5-4165f8294b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4400e7-7bpsm" Nov 03 14:39:24.776702 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.776665 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e634e8b3-2f74-482d-aaf5-4165f8294b45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf4400e7-7bpsm\" (UID: \"e634e8b3-2f74-482d-aaf5-4165f8294b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4400e7-7bpsm" Nov 03 14:39:24.776755 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.776724 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e634e8b3-2f74-482d-aaf5-4165f8294b45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf4400e7-7bpsm\" (UID: \"e634e8b3-2f74-482d-aaf5-4165f8294b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4400e7-7bpsm" Nov 03 14:39:24.878338 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.878306 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xqs4\" (UniqueName: \"kubernetes.io/projected/e634e8b3-2f74-482d-aaf5-4165f8294b45-kube-api-access-8xqs4\") pod \"maas-api-revoke-sub-cf4400e7-7bpsm\" (UID: \"e634e8b3-2f74-482d-aaf5-4165f8294b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4400e7-7bpsm" Nov 03 14:39:24.878525 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.878367 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e634e8b3-2f74-482d-aaf5-4165f8294b45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf4400e7-7bpsm\" (UID: \"e634e8b3-2f74-482d-aaf5-4165f8294b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4400e7-7bpsm" Nov 03 14:39:24.878525 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.878396 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e634e8b3-2f74-482d-aaf5-4165f8294b45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf4400e7-7bpsm\" (UID: \"e634e8b3-2f74-482d-aaf5-4165f8294b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4400e7-7bpsm" Nov 03 14:39:24.879081 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.879059 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e634e8b3-2f74-482d-aaf5-4165f8294b45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf4400e7-7bpsm\" (UID: \"e634e8b3-2f74-482d-aaf5-4165f8294b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4400e7-7bpsm" Nov 03 14:39:24.880752 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.880730 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e634e8b3-2f74-482d-aaf5-4165f8294b45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf4400e7-7bpsm\" (UID: \"e634e8b3-2f74-482d-aaf5-4165f8294b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4400e7-7bpsm" Nov 03 14:39:24.901712 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:24.901691 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xqs4\" (UniqueName: \"kubernetes.io/projected/e634e8b3-2f74-482d-aaf5-4165f8294b45-kube-api-access-8xqs4\") pod \"maas-api-revoke-sub-cf4400e7-7bpsm\" (UID: \"e634e8b3-2f74-482d-aaf5-4165f8294b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4400e7-7bpsm" Nov 03 14:39:25.016130 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:25.016098 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4400e7-7bpsm" Nov 03 14:39:25.326591 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:25.326557 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf4400e7-7bpsm"] Nov 03 14:39:25.327776 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:39:25.327750 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode634e8b3_2f74_482d_aaf5_4165f8294b45.slice/crio-504eef31bf5cdf18c2ed5b174def5964ebfe640aa67634a7aaf80139833a90d2 WatchSource:0}: Error finding container 504eef31bf5cdf18c2ed5b174def5964ebfe640aa67634a7aaf80139833a90d2: Status 404 returned error can't find the container with id 504eef31bf5cdf18c2ed5b174def5964ebfe640aa67634a7aaf80139833a90d2 Nov 03 14:39:25.442809 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:25.442769 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4400e7-7bpsm" event={"ID":"e634e8b3-2f74-482d-aaf5-4165f8294b45","Type":"ContainerStarted","Data":"504eef31bf5cdf18c2ed5b174def5964ebfe640aa67634a7aaf80139833a90d2"} Nov 03 14:39:26.452186 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:26.452097 5060 generic.go:356] "Generic (PLEG): container finished" podID="e634e8b3-2f74-482d-aaf5-4165f8294b45" containerID="51c1decf449affacb7440685a1ef6d4efa2e33b66ad244ca702fc45a34062ac6" exitCode=0 Nov 03 14:39:26.452591 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:26.452193 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4400e7-7bpsm" event={"ID":"e634e8b3-2f74-482d-aaf5-4165f8294b45","Type":"ContainerDied","Data":"51c1decf449affacb7440685a1ef6d4efa2e33b66ad244ca702fc45a34062ac6"} Nov 03 14:39:27.618889 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:27.618863 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4400e7-7bpsm" Nov 03 14:39:27.710056 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:27.709961 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e634e8b3-2f74-482d-aaf5-4165f8294b45-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e634e8b3-2f74-482d-aaf5-4165f8294b45-maas-api-xks-ca\") pod \"e634e8b3-2f74-482d-aaf5-4165f8294b45\" (UID: \"e634e8b3-2f74-482d-aaf5-4165f8294b45\") " Nov 03 14:39:27.710056 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:27.710030 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e634e8b3-2f74-482d-aaf5-4165f8294b45-kube-api-access-8xqs4\" (UniqueName: \"kubernetes.io/projected/e634e8b3-2f74-482d-aaf5-4165f8294b45-kube-api-access-8xqs4\") pod \"e634e8b3-2f74-482d-aaf5-4165f8294b45\" (UID: \"e634e8b3-2f74-482d-aaf5-4165f8294b45\") " Nov 03 14:39:27.710304 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:27.710188 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e634e8b3-2f74-482d-aaf5-4165f8294b45-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e634e8b3-2f74-482d-aaf5-4165f8294b45-maas-api-openshift-ca\") pod \"e634e8b3-2f74-482d-aaf5-4165f8294b45\" (UID: \"e634e8b3-2f74-482d-aaf5-4165f8294b45\") " Nov 03 14:39:27.710636 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:27.710595 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e634e8b3-2f74-482d-aaf5-4165f8294b45-maas-api-openshift-ca" pod "e634e8b3-2f74-482d-aaf5-4165f8294b45" (UID: "e634e8b3-2f74-482d-aaf5-4165f8294b45"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:39:27.712293 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:27.712266 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e634e8b3-2f74-482d-aaf5-4165f8294b45-maas-api-xks-ca" pod "e634e8b3-2f74-482d-aaf5-4165f8294b45" (UID: "e634e8b3-2f74-482d-aaf5-4165f8294b45"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:39:27.712686 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:27.712662 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e634e8b3-2f74-482d-aaf5-4165f8294b45-kube-api-access-8xqs4" pod "e634e8b3-2f74-482d-aaf5-4165f8294b45" (UID: "e634e8b3-2f74-482d-aaf5-4165f8294b45"). InnerVolumeSpecName "kube-api-access-8xqs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:39:27.811616 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:27.811577 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e634e8b3-2f74-482d-aaf5-4165f8294b45-maas-api-xks-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:39:27.811616 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:27.811609 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xqs4\" (UniqueName: \"kubernetes.io/projected/e634e8b3-2f74-482d-aaf5-4165f8294b45-kube-api-access-8xqs4\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:39:27.811616 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:27.811621 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e634e8b3-2f74-482d-aaf5-4165f8294b45-maas-api-openshift-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:39:28.464929 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:28.464897 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4400e7-7bpsm" Nov 03 14:39:28.464929 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:28.464923 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4400e7-7bpsm" event={"ID":"e634e8b3-2f74-482d-aaf5-4165f8294b45","Type":"ContainerDied","Data":"504eef31bf5cdf18c2ed5b174def5964ebfe640aa67634a7aaf80139833a90d2"} Nov 03 14:39:28.465130 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:28.464967 5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="504eef31bf5cdf18c2ed5b174def5964ebfe640aa67634a7aaf80139833a90d2" Nov 03 14:39:28.523513 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:28.523416 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8"] Nov 03 14:39:28.523690 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:28.523611 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" podUID="a88d21ce-086a-414a-911a-5561e2cb8023" containerName="istio-proxy" containerID="cri-o://ee16f02a8ea2bfaf37abb4dfb62eddb03226e4762ce4d44ab6356921a3528b84" gracePeriod=30 Nov 03 14:39:29.187015 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:29.186981 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-156ae1/distinct-2-w3-156ae1-kserve-5988894d76-vsdlm"] Nov 03 14:39:29.195524 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:29.195500 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98"] Nov 03 14:39:33.896432 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:33.896405 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" Nov 03 14:39:33.988192 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:33.988154 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a88d21ce-086a-414a-911a-5561e2cb8023-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a88d21ce-086a-414a-911a-5561e2cb8023-credential-socket\") pod \"a88d21ce-086a-414a-911a-5561e2cb8023\" (UID: \"a88d21ce-086a-414a-911a-5561e2cb8023\") " Nov 03 14:39:33.988192 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:33.988198 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a88d21ce-086a-414a-911a-5561e2cb8023-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a88d21ce-086a-414a-911a-5561e2cb8023-istio-podinfo\") pod \"a88d21ce-086a-414a-911a-5561e2cb8023\" (UID: \"a88d21ce-086a-414a-911a-5561e2cb8023\") " Nov 03 14:39:33.988428 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:33.988231 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a88d21ce-086a-414a-911a-5561e2cb8023-istio-token\" (UniqueName: \"kubernetes.io/projected/a88d21ce-086a-414a-911a-5561e2cb8023-istio-token\") pod \"a88d21ce-086a-414a-911a-5561e2cb8023\" (UID: \"a88d21ce-086a-414a-911a-5561e2cb8023\") " Nov 03 14:39:33.988428 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:33.988282 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a88d21ce-086a-414a-911a-5561e2cb8023-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a88d21ce-086a-414a-911a-5561e2cb8023-istio-envoy\") pod \"a88d21ce-086a-414a-911a-5561e2cb8023\" (UID: \"a88d21ce-086a-414a-911a-5561e2cb8023\") " Nov 03 14:39:33.988428 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:33.988320 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a88d21ce-086a-414a-911a-5561e2cb8023-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a88d21ce-086a-414a-911a-5561e2cb8023-workload-certs\") pod \"a88d21ce-086a-414a-911a-5561e2cb8023\" (UID: \"a88d21ce-086a-414a-911a-5561e2cb8023\") " Nov 03 14:39:33.988428 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:33.988356 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a88d21ce-086a-414a-911a-5561e2cb8023-kube-api-access-tms5j\" (UniqueName: \"kubernetes.io/projected/a88d21ce-086a-414a-911a-5561e2cb8023-kube-api-access-tms5j\") pod \"a88d21ce-086a-414a-911a-5561e2cb8023\" (UID: \"a88d21ce-086a-414a-911a-5561e2cb8023\") " Nov 03 14:39:33.988428 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:33.988412 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a88d21ce-086a-414a-911a-5561e2cb8023-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a88d21ce-086a-414a-911a-5561e2cb8023-istio-data\") pod \"a88d21ce-086a-414a-911a-5561e2cb8023\" (UID: \"a88d21ce-086a-414a-911a-5561e2cb8023\") " Nov 03 14:39:33.988709 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:33.988453 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a88d21ce-086a-414a-911a-5561e2cb8023-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a88d21ce-086a-414a-911a-5561e2cb8023-istiod-ca-cert\") pod \"a88d21ce-086a-414a-911a-5561e2cb8023\" (UID: \"a88d21ce-086a-414a-911a-5561e2cb8023\") " Nov 03 14:39:33.988709 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:33.988472 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a88d21ce-086a-414a-911a-5561e2cb8023-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a88d21ce-086a-414a-911a-5561e2cb8023-workload-socket\") pod \"a88d21ce-086a-414a-911a-5561e2cb8023\" (UID: \"a88d21ce-086a-414a-911a-5561e2cb8023\") " Nov 03 14:39:33.988972 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:33.988943 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a88d21ce-086a-414a-911a-5561e2cb8023-workload-socket" pod "a88d21ce-086a-414a-911a-5561e2cb8023" (UID: "a88d21ce-086a-414a-911a-5561e2cb8023"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:39:33.989227 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:33.989198 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a88d21ce-086a-414a-911a-5561e2cb8023-credential-socket" pod "a88d21ce-086a-414a-911a-5561e2cb8023" (UID: "a88d21ce-086a-414a-911a-5561e2cb8023"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:39:33.989227 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:33.989214 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a88d21ce-086a-414a-911a-5561e2cb8023-istio-data" pod "a88d21ce-086a-414a-911a-5561e2cb8023" (UID: "a88d21ce-086a-414a-911a-5561e2cb8023"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:39:33.989355 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:33.989333 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a88d21ce-086a-414a-911a-5561e2cb8023-workload-certs" pod "a88d21ce-086a-414a-911a-5561e2cb8023" (UID: "a88d21ce-086a-414a-911a-5561e2cb8023"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:39:33.989509 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:33.989477 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a88d21ce-086a-414a-911a-5561e2cb8023-istiod-ca-cert" pod "a88d21ce-086a-414a-911a-5561e2cb8023" (UID: "a88d21ce-086a-414a-911a-5561e2cb8023"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:39:33.991181 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:33.991160 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a88d21ce-086a-414a-911a-5561e2cb8023-istio-podinfo" pod "a88d21ce-086a-414a-911a-5561e2cb8023" (UID: "a88d21ce-086a-414a-911a-5561e2cb8023"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:39:33.991785 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:33.991758 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a88d21ce-086a-414a-911a-5561e2cb8023-kube-api-access-tms5j" pod "a88d21ce-086a-414a-911a-5561e2cb8023" (UID: "a88d21ce-086a-414a-911a-5561e2cb8023"). InnerVolumeSpecName "kube-api-access-tms5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:39:33.991785 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:33.991771 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a88d21ce-086a-414a-911a-5561e2cb8023-istio-token" pod "a88d21ce-086a-414a-911a-5561e2cb8023" (UID: "a88d21ce-086a-414a-911a-5561e2cb8023"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:39:33.991967 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:33.991912 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a88d21ce-086a-414a-911a-5561e2cb8023-istio-envoy" pod "a88d21ce-086a-414a-911a-5561e2cb8023" (UID: "a88d21ce-086a-414a-911a-5561e2cb8023"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:39:34.090250 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:34.090207 5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a88d21ce-086a-414a-911a-5561e2cb8023-istiod-ca-cert\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:39:34.090250 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:34.090245 5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a88d21ce-086a-414a-911a-5561e2cb8023-workload-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:39:34.090250 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:34.090258 5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a88d21ce-086a-414a-911a-5561e2cb8023-credential-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:39:34.090499 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:34.090271 5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a88d21ce-086a-414a-911a-5561e2cb8023-istio-podinfo\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:39:34.090499 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:34.090284 5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a88d21ce-086a-414a-911a-5561e2cb8023-istio-token\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:39:34.090499 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:34.090296 5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a88d21ce-086a-414a-911a-5561e2cb8023-istio-envoy\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:39:34.090499 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:34.090308 5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a88d21ce-086a-414a-911a-5561e2cb8023-workload-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:39:34.090499 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:34.090321 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tms5j\" (UniqueName: \"kubernetes.io/projected/a88d21ce-086a-414a-911a-5561e2cb8023-kube-api-access-tms5j\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:39:34.090499 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:34.090332 5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a88d21ce-086a-414a-911a-5561e2cb8023-istio-data\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:39:34.503040 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:34.502996 5060 generic.go:356] "Generic (PLEG): container finished" podID="a88d21ce-086a-414a-911a-5561e2cb8023" containerID="ee16f02a8ea2bfaf37abb4dfb62eddb03226e4762ce4d44ab6356921a3528b84" exitCode=0 Nov 03 14:39:34.503203 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:34.503054 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" event={"ID":"a88d21ce-086a-414a-911a-5561e2cb8023","Type":"ContainerDied","Data":"ee16f02a8ea2bfaf37abb4dfb62eddb03226e4762ce4d44ab6356921a3528b84"} Nov 03 14:39:34.503203 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:34.503104 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" event={"ID":"a88d21ce-086a-414a-911a-5561e2cb8023","Type":"ContainerDied","Data":"09d8fb217c248d63a80c981d291aceb2b5a68e20a47167d66006b61371516931"} Nov 03 14:39:34.503203 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:34.503132 5060 scope.go:122] "RemoveContainer" containerID="ee16f02a8ea2bfaf37abb4dfb62eddb03226e4762ce4d44ab6356921a3528b84" Nov 03 14:39:34.503203 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:34.503055 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" Nov 03 14:39:34.512504 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:34.512487 5060 scope.go:122] "RemoveContainer" containerID="ee16f02a8ea2bfaf37abb4dfb62eddb03226e4762ce4d44ab6356921a3528b84" Nov 03 14:39:34.512768 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:39:34.512751 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee16f02a8ea2bfaf37abb4dfb62eddb03226e4762ce4d44ab6356921a3528b84\": container with ID starting with ee16f02a8ea2bfaf37abb4dfb62eddb03226e4762ce4d44ab6356921a3528b84 not found: ID does not exist" containerID="ee16f02a8ea2bfaf37abb4dfb62eddb03226e4762ce4d44ab6356921a3528b84" Nov 03 14:39:34.512807 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:34.512777 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee16f02a8ea2bfaf37abb4dfb62eddb03226e4762ce4d44ab6356921a3528b84"} err="failed to get container status \"ee16f02a8ea2bfaf37abb4dfb62eddb03226e4762ce4d44ab6356921a3528b84\": rpc error: code = NotFound desc = could not find container \"ee16f02a8ea2bfaf37abb4dfb62eddb03226e4762ce4d44ab6356921a3528b84\": container with ID starting with ee16f02a8ea2bfaf37abb4dfb62eddb03226e4762ce4d44ab6356921a3528b84 not found: ID does not exist" Nov 03 14:39:34.576964 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:34.576940 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8"] Nov 03 14:39:34.584091 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:34.584065 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8"] Nov 03 14:39:34.773700 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:34.773618 5060 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:39:34.773700 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:34.773679 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-bp5q8" podUID="a88d21ce-086a-414a-911a-5561e2cb8023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:39:36.249541 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:36.249508 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a88d21ce-086a-414a-911a-5561e2cb8023" path="/var/lib/kubelet/pods/a88d21ce-086a-414a-911a-5561e2cb8023/volumes" Nov 03 14:39:39.442617 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:39:39.442573 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:39:44.207470 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.207434 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-156ae1/distinct-2-w3-156ae1-kserve-5988894d76-vsdlm" podUID="59dcdef3-f88e-4463-a0dd-1917e915a516" containerName="main" containerID="cri-o://b6f58148cd1a1713b2b050d10667987ec0f3bb884f5a9a08eb38c9dd50c90ad9" gracePeriod=45 Nov 03 14:39:44.214598 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.214568 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98" podUID="3aaaeea4-134f-4d17-a180-b4857c6425d3" containerName="main" containerID="cri-o://ea006e5bfb0cf3fca5446ad0f429c1bcc9b3d0e0c98c79459a13e68ecd12bbc6" gracePeriod=45 Nov 03 14:39:44.537992 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.537958 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-156ae1/distinct-2-w3-156ae1-kserve-5988894d76-vsdlm" Nov 03 14:39:44.540286 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.540247 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98" Nov 03 14:39:44.567649 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.567616 5060 generic.go:356] "Generic (PLEG): container finished" podID="3aaaeea4-134f-4d17-a180-b4857c6425d3" containerID="ea006e5bfb0cf3fca5446ad0f429c1bcc9b3d0e0c98c79459a13e68ecd12bbc6" exitCode=0 Nov 03 14:39:44.567649 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.567654 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98" Nov 03 14:39:44.567903 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.567697 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98" event={"ID":"3aaaeea4-134f-4d17-a180-b4857c6425d3","Type":"ContainerDied","Data":"ea006e5bfb0cf3fca5446ad0f429c1bcc9b3d0e0c98c79459a13e68ecd12bbc6"} Nov 03 14:39:44.567903 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.567736 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98" event={"ID":"3aaaeea4-134f-4d17-a180-b4857c6425d3","Type":"ContainerDied","Data":"4581efed9093a0967c17779cde329fd164ca71bace7efb07be66a871cdec0362"} Nov 03 14:39:44.567903 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.567763 5060 scope.go:122] "RemoveContainer" containerID="ea006e5bfb0cf3fca5446ad0f429c1bcc9b3d0e0c98c79459a13e68ecd12bbc6" Nov 03 14:39:44.569390 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.569367 5060 generic.go:356] "Generic (PLEG): container finished" podID="59dcdef3-f88e-4463-a0dd-1917e915a516" containerID="b6f58148cd1a1713b2b050d10667987ec0f3bb884f5a9a08eb38c9dd50c90ad9" exitCode=0 Nov 03 14:39:44.569503 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.569418 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-156ae1/distinct-2-w3-156ae1-kserve-5988894d76-vsdlm" Nov 03 14:39:44.569503 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.569449 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-156ae1/distinct-2-w3-156ae1-kserve-5988894d76-vsdlm" event={"ID":"59dcdef3-f88e-4463-a0dd-1917e915a516","Type":"ContainerDied","Data":"b6f58148cd1a1713b2b050d10667987ec0f3bb884f5a9a08eb38c9dd50c90ad9"} Nov 03 14:39:44.569503 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.569481 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-156ae1/distinct-2-w3-156ae1-kserve-5988894d76-vsdlm" event={"ID":"59dcdef3-f88e-4463-a0dd-1917e915a516","Type":"ContainerDied","Data":"70479f9753393fdc2f096e381f9f395a6df94e8c97cc8499ec88263d059ef830"} Nov 03 14:39:44.579545 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.579526 5060 scope.go:122] "RemoveContainer" containerID="ea006e5bfb0cf3fca5446ad0f429c1bcc9b3d0e0c98c79459a13e68ecd12bbc6" Nov 03 14:39:44.579825 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:39:44.579803 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea006e5bfb0cf3fca5446ad0f429c1bcc9b3d0e0c98c79459a13e68ecd12bbc6\": container with ID starting with ea006e5bfb0cf3fca5446ad0f429c1bcc9b3d0e0c98c79459a13e68ecd12bbc6 not found: ID does not exist" containerID="ea006e5bfb0cf3fca5446ad0f429c1bcc9b3d0e0c98c79459a13e68ecd12bbc6" Nov 03 14:39:44.579933 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.579880 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea006e5bfb0cf3fca5446ad0f429c1bcc9b3d0e0c98c79459a13e68ecd12bbc6"} err="failed to get container status \"ea006e5bfb0cf3fca5446ad0f429c1bcc9b3d0e0c98c79459a13e68ecd12bbc6\": rpc error: code = NotFound desc = could not find container \"ea006e5bfb0cf3fca5446ad0f429c1bcc9b3d0e0c98c79459a13e68ecd12bbc6\": container with ID starting with ea006e5bfb0cf3fca5446ad0f429c1bcc9b3d0e0c98c79459a13e68ecd12bbc6 not found: ID does not exist" Nov 03 14:39:44.579933 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.579903 5060 scope.go:122] "RemoveContainer" containerID="b6f58148cd1a1713b2b050d10667987ec0f3bb884f5a9a08eb38c9dd50c90ad9" Nov 03 14:39:44.591914 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.591888 5060 scope.go:122] "RemoveContainer" containerID="b6f58148cd1a1713b2b050d10667987ec0f3bb884f5a9a08eb38c9dd50c90ad9" Nov 03 14:39:44.594340 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:39:44.594305 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6f58148cd1a1713b2b050d10667987ec0f3bb884f5a9a08eb38c9dd50c90ad9\": container with ID starting with b6f58148cd1a1713b2b050d10667987ec0f3bb884f5a9a08eb38c9dd50c90ad9 not found: ID does not exist" containerID="b6f58148cd1a1713b2b050d10667987ec0f3bb884f5a9a08eb38c9dd50c90ad9" Nov 03 14:39:44.594499 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.594346 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6f58148cd1a1713b2b050d10667987ec0f3bb884f5a9a08eb38c9dd50c90ad9"} err="failed to get container status \"b6f58148cd1a1713b2b050d10667987ec0f3bb884f5a9a08eb38c9dd50c90ad9\": rpc error: code = NotFound desc = could not find container \"b6f58148cd1a1713b2b050d10667987ec0f3bb884f5a9a08eb38c9dd50c90ad9\": container with ID starting with b6f58148cd1a1713b2b050d10667987ec0f3bb884f5a9a08eb38c9dd50c90ad9 not found: ID does not exist" Nov 03 14:39:44.613478 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.613444 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aaaeea4-134f-4d17-a180-b4857c6425d3-home\" (UniqueName: \"kubernetes.io/empty-dir/3aaaeea4-134f-4d17-a180-b4857c6425d3-home\") pod \"3aaaeea4-134f-4d17-a180-b4857c6425d3\" (UID: \"3aaaeea4-134f-4d17-a180-b4857c6425d3\") " Nov 03 14:39:44.613788 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.613758 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aaaeea4-134f-4d17-a180-b4857c6425d3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3aaaeea4-134f-4d17-a180-b4857c6425d3-dshm\") pod \"3aaaeea4-134f-4d17-a180-b4857c6425d3\" (UID: \"3aaaeea4-134f-4d17-a180-b4857c6425d3\") " Nov 03 14:39:44.613943 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.613921 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59dcdef3-f88e-4463-a0dd-1917e915a516-tls-certs\" (UniqueName: \"kubernetes.io/secret/59dcdef3-f88e-4463-a0dd-1917e915a516-tls-certs\") pod \"59dcdef3-f88e-4463-a0dd-1917e915a516\" (UID: \"59dcdef3-f88e-4463-a0dd-1917e915a516\") " Nov 03 14:39:44.614014 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.613964 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59dcdef3-f88e-4463-a0dd-1917e915a516-dshm\" (UniqueName: \"kubernetes.io/empty-dir/59dcdef3-f88e-4463-a0dd-1917e915a516-dshm\") pod \"59dcdef3-f88e-4463-a0dd-1917e915a516\" (UID: \"59dcdef3-f88e-4463-a0dd-1917e915a516\") " Nov 03 14:39:44.614014 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.614000 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59dcdef3-f88e-4463-a0dd-1917e915a516-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59dcdef3-f88e-4463-a0dd-1917e915a516-tmp-dir\") pod \"59dcdef3-f88e-4463-a0dd-1917e915a516\" (UID: \"59dcdef3-f88e-4463-a0dd-1917e915a516\") " Nov 03 14:39:44.614130 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.614037 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aaaeea4-134f-4d17-a180-b4857c6425d3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3aaaeea4-134f-4d17-a180-b4857c6425d3-tmp-dir\") pod \"3aaaeea4-134f-4d17-a180-b4857c6425d3\" (UID: \"3aaaeea4-134f-4d17-a180-b4857c6425d3\") " Nov 03 14:39:44.614130 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.614088 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3aaaeea4-134f-4d17-a180-b4857c6425d3-tls-certs\" (UniqueName: \"kubernetes.io/secret/3aaaeea4-134f-4d17-a180-b4857c6425d3-tls-certs\") pod \"3aaaeea4-134f-4d17-a180-b4857c6425d3\" (UID: \"3aaaeea4-134f-4d17-a180-b4857c6425d3\") " Nov 03 14:39:44.614130 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.614119 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59dcdef3-f88e-4463-a0dd-1917e915a516-kube-api-access-ggq2x\" (UniqueName: \"kubernetes.io/projected/59dcdef3-f88e-4463-a0dd-1917e915a516-kube-api-access-ggq2x\") pod \"59dcdef3-f88e-4463-a0dd-1917e915a516\" (UID: \"59dcdef3-f88e-4463-a0dd-1917e915a516\") " Nov 03 14:39:44.614283 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.614148 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aaaeea4-134f-4d17-a180-b4857c6425d3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3aaaeea4-134f-4d17-a180-b4857c6425d3-model-cache\") pod \"3aaaeea4-134f-4d17-a180-b4857c6425d3\" (UID: \"3aaaeea4-134f-4d17-a180-b4857c6425d3\") " Nov 03 14:39:44.614283 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.614222 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59dcdef3-f88e-4463-a0dd-1917e915a516-home\" (UniqueName: \"kubernetes.io/empty-dir/59dcdef3-f88e-4463-a0dd-1917e915a516-home\") pod \"59dcdef3-f88e-4463-a0dd-1917e915a516\" (UID: \"59dcdef3-f88e-4463-a0dd-1917e915a516\") " Nov 03 14:39:44.614283 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.614260 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aaaeea4-134f-4d17-a180-b4857c6425d3-kube-api-access-lrcf8\" (UniqueName: \"kubernetes.io/projected/3aaaeea4-134f-4d17-a180-b4857c6425d3-kube-api-access-lrcf8\") pod \"3aaaeea4-134f-4d17-a180-b4857c6425d3\" (UID: \"3aaaeea4-134f-4d17-a180-b4857c6425d3\") " Nov 03 14:39:44.614452 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.614359 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59dcdef3-f88e-4463-a0dd-1917e915a516-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59dcdef3-f88e-4463-a0dd-1917e915a516-model-cache\") pod \"59dcdef3-f88e-4463-a0dd-1917e915a516\" (UID: \"59dcdef3-f88e-4463-a0dd-1917e915a516\") " Nov 03 14:39:44.615057 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.614878 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59dcdef3-f88e-4463-a0dd-1917e915a516-home" pod "59dcdef3-f88e-4463-a0dd-1917e915a516" (UID: "59dcdef3-f88e-4463-a0dd-1917e915a516"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:39:44.615179 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.615132 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aaaeea4-134f-4d17-a180-b4857c6425d3-home" pod "3aaaeea4-134f-4d17-a180-b4857c6425d3" (UID: "3aaaeea4-134f-4d17-a180-b4857c6425d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:39:44.615595 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.615350 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59dcdef3-f88e-4463-a0dd-1917e915a516-tmp-dir" pod "59dcdef3-f88e-4463-a0dd-1917e915a516" (UID: "59dcdef3-f88e-4463-a0dd-1917e915a516"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:39:44.615595 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.615198 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aaaeea4-134f-4d17-a180-b4857c6425d3-model-cache" pod "3aaaeea4-134f-4d17-a180-b4857c6425d3" (UID: "3aaaeea4-134f-4d17-a180-b4857c6425d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:39:44.615595 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.615568 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59dcdef3-f88e-4463-a0dd-1917e915a516-model-cache" pod "59dcdef3-f88e-4463-a0dd-1917e915a516" (UID: "59dcdef3-f88e-4463-a0dd-1917e915a516"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:39:44.615595 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.615562 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aaaeea4-134f-4d17-a180-b4857c6425d3-tmp-dir" pod "3aaaeea4-134f-4d17-a180-b4857c6425d3" (UID: "3aaaeea4-134f-4d17-a180-b4857c6425d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:39:44.617602 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.617576 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59dcdef3-f88e-4463-a0dd-1917e915a516-tls-certs" pod "59dcdef3-f88e-4463-a0dd-1917e915a516" (UID: "59dcdef3-f88e-4463-a0dd-1917e915a516"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:39:44.626358 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.626324 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aaaeea4-134f-4d17-a180-b4857c6425d3-kube-api-access-lrcf8" pod "3aaaeea4-134f-4d17-a180-b4857c6425d3" (UID: "3aaaeea4-134f-4d17-a180-b4857c6425d3"). InnerVolumeSpecName "kube-api-access-lrcf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:39:44.626529 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.626379 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59dcdef3-f88e-4463-a0dd-1917e915a516-dshm" pod "59dcdef3-f88e-4463-a0dd-1917e915a516" (UID: "59dcdef3-f88e-4463-a0dd-1917e915a516"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:39:44.626661 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.626375 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aaaeea4-134f-4d17-a180-b4857c6425d3-dshm" pod "3aaaeea4-134f-4d17-a180-b4857c6425d3" (UID: "3aaaeea4-134f-4d17-a180-b4857c6425d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:39:44.626793 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.626406 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59dcdef3-f88e-4463-a0dd-1917e915a516-kube-api-access-ggq2x" pod "59dcdef3-f88e-4463-a0dd-1917e915a516" (UID: "59dcdef3-f88e-4463-a0dd-1917e915a516"). InnerVolumeSpecName "kube-api-access-ggq2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:39:44.628052 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.628024 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3aaaeea4-134f-4d17-a180-b4857c6425d3-tls-certs" pod "3aaaeea4-134f-4d17-a180-b4857c6425d3" (UID: "3aaaeea4-134f-4d17-a180-b4857c6425d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:39:44.716002 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.715962 5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3aaaeea4-134f-4d17-a180-b4857c6425d3-tls-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:39:44.716002 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.716000 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggq2x\" (UniqueName: \"kubernetes.io/projected/59dcdef3-f88e-4463-a0dd-1917e915a516-kube-api-access-ggq2x\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:39:44.716002 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.716011 5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3aaaeea4-134f-4d17-a180-b4857c6425d3-model-cache\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:39:44.716256 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.716020 5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59dcdef3-f88e-4463-a0dd-1917e915a516-home\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:39:44.716256 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.716029 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrcf8\" (UniqueName: \"kubernetes.io/projected/3aaaeea4-134f-4d17-a180-b4857c6425d3-kube-api-access-lrcf8\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:39:44.716256 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.716037 5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59dcdef3-f88e-4463-a0dd-1917e915a516-model-cache\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:39:44.716256 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.716050 5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3aaaeea4-134f-4d17-a180-b4857c6425d3-home\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:39:44.716256 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.716061 5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3aaaeea4-134f-4d17-a180-b4857c6425d3-dshm\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:39:44.716256 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.716068 5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59dcdef3-f88e-4463-a0dd-1917e915a516-tls-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:39:44.716256 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.716075 5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59dcdef3-f88e-4463-a0dd-1917e915a516-dshm\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:39:44.716256 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.716082 5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59dcdef3-f88e-4463-a0dd-1917e915a516-tmp-dir\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:39:44.716256 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.716090 5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3aaaeea4-134f-4d17-a180-b4857c6425d3-tmp-dir\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:39:44.922468 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.922429 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98"] Nov 03 14:39:44.927349 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.927322 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98"] Nov 03 14:39:44.967218 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.967185 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-156ae1/distinct-2-w3-156ae1-kserve-5988894d76-vsdlm"] Nov 03 14:39:44.970589 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:44.970549 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-156ae1/distinct-2-w3-156ae1-kserve-5988894d76-vsdlm"] Nov 03 14:39:45.422675 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:45.422622 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/distinct-w3-156ae1-kserve-865cd6c59f-8rc98" podUID="3aaaeea4-134f-4d17-a180-b4857c6425d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 14:39:46.248776 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:46.248733 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aaaeea4-134f-4d17-a180-b4857c6425d3" path="/var/lib/kubelet/pods/3aaaeea4-134f-4d17-a180-b4857c6425d3/volumes" Nov 03 14:39:46.249264 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:39:46.249241 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59dcdef3-f88e-4463-a0dd-1917e915a516" path="/var/lib/kubelet/pods/59dcdef3-f88e-4463-a0dd-1917e915a516/volumes" Nov 03 14:40:09.526888 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:40:09.517095 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:40:11.819936 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:11.819897 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz"] Nov 03 14:40:11.820742 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:11.820681 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" podUID="b3327f34-66f3-4a70-915a-aa5ad8685fad" containerName="istio-proxy" containerID="cri-o://42e4fcc0f5d87a398aeeabbebcddac52534e1612a7107b5f0f20c1d3b29b20a4" gracePeriod=30 Nov 03 14:40:11.822659 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:11.822632 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v"] Nov 03 14:40:11.822981 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:11.822956 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" podUID="f6695b0e-c9b6-4119-a0b2-508b212a3ece" containerName="istio-proxy" containerID="cri-o://b6f1d2902246456c639f4b454edd5fbb2ddab0a2d1c3474d794c7fdcb26b6673" gracePeriod=30 Nov 03 14:40:17.141046 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.141012 5060 generic.go:356] "Generic (PLEG): container finished" podID="f6695b0e-c9b6-4119-a0b2-508b212a3ece" containerID="b6f1d2902246456c639f4b454edd5fbb2ddab0a2d1c3474d794c7fdcb26b6673" exitCode=0 Nov 03 14:40:17.141543 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.141227 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" event={"ID":"f6695b0e-c9b6-4119-a0b2-508b212a3ece","Type":"ContainerDied","Data":"b6f1d2902246456c639f4b454edd5fbb2ddab0a2d1c3474d794c7fdcb26b6673"} Nov 03 14:40:17.143411 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.143381 5060 generic.go:356] "Generic (PLEG): container finished" podID="b3327f34-66f3-4a70-915a-aa5ad8685fad" containerID="42e4fcc0f5d87a398aeeabbebcddac52534e1612a7107b5f0f20c1d3b29b20a4" exitCode=0 Nov 03 14:40:17.143530 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.143439 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" event={"ID":"b3327f34-66f3-4a70-915a-aa5ad8685fad","Type":"ContainerDied","Data":"42e4fcc0f5d87a398aeeabbebcddac52534e1612a7107b5f0f20c1d3b29b20a4"} Nov 03 14:40:17.187668 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.187647 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" Nov 03 14:40:17.204761 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.204735 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" Nov 03 14:40:17.285081 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.285045 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6695b0e-c9b6-4119-a0b2-508b212a3ece-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6695b0e-c9b6-4119-a0b2-508b212a3ece-istio-podinfo\") pod \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\" (UID: \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\") " Nov 03 14:40:17.285272 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.285106 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6695b0e-c9b6-4119-a0b2-508b212a3ece-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6695b0e-c9b6-4119-a0b2-508b212a3ece-credential-socket\") pod \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\" (UID: \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\") " Nov 03 14:40:17.285272 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.285140 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3327f34-66f3-4a70-915a-aa5ad8685fad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3327f34-66f3-4a70-915a-aa5ad8685fad-istio-envoy\") pod \"b3327f34-66f3-4a70-915a-aa5ad8685fad\" (UID: \"b3327f34-66f3-4a70-915a-aa5ad8685fad\") " Nov 03 14:40:17.285272 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.285177 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3327f34-66f3-4a70-915a-aa5ad8685fad-kube-api-access-k5kq5\" (UniqueName: \"kubernetes.io/projected/b3327f34-66f3-4a70-915a-aa5ad8685fad-kube-api-access-k5kq5\") pod \"b3327f34-66f3-4a70-915a-aa5ad8685fad\" (UID: \"b3327f34-66f3-4a70-915a-aa5ad8685fad\") " Nov 03 14:40:17.285272 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.285216 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6695b0e-c9b6-4119-a0b2-508b212a3ece-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6695b0e-c9b6-4119-a0b2-508b212a3ece-istio-envoy\") pod \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\" (UID: \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\") " Nov 03 14:40:17.285507 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.285370 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3327f34-66f3-4a70-915a-aa5ad8685fad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3327f34-66f3-4a70-915a-aa5ad8685fad-istio-podinfo\") pod \"b3327f34-66f3-4a70-915a-aa5ad8685fad\" (UID: \"b3327f34-66f3-4a70-915a-aa5ad8685fad\") " Nov 03 14:40:17.285507 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.285443 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6695b0e-c9b6-4119-a0b2-508b212a3ece-istio-token\" (UniqueName: \"kubernetes.io/projected/f6695b0e-c9b6-4119-a0b2-508b212a3ece-istio-token\") pod \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\" (UID: \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\") " Nov 03 14:40:17.285629 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.285450 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6695b0e-c9b6-4119-a0b2-508b212a3ece-credential-socket" pod "f6695b0e-c9b6-4119-a0b2-508b212a3ece" (UID: "f6695b0e-c9b6-4119-a0b2-508b212a3ece"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:40:17.285629 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.285526 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6695b0e-c9b6-4119-a0b2-508b212a3ece-kube-api-access-v4hq6\" (UniqueName: \"kubernetes.io/projected/f6695b0e-c9b6-4119-a0b2-508b212a3ece-kube-api-access-v4hq6\") pod \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\" (UID: \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\") " Nov 03 14:40:17.286024 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.285952 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3327f34-66f3-4a70-915a-aa5ad8685fad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3327f34-66f3-4a70-915a-aa5ad8685fad-workload-certs\") pod \"b3327f34-66f3-4a70-915a-aa5ad8685fad\" (UID: \"b3327f34-66f3-4a70-915a-aa5ad8685fad\") " Nov 03 14:40:17.286340 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.286270 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6695b0e-c9b6-4119-a0b2-508b212a3ece-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6695b0e-c9b6-4119-a0b2-508b212a3ece-workload-certs\") pod \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\" (UID: \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\") " Nov 03 14:40:17.286578 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.286539 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3327f34-66f3-4a70-915a-aa5ad8685fad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3327f34-66f3-4a70-915a-aa5ad8685fad-credential-socket\") pod \"b3327f34-66f3-4a70-915a-aa5ad8685fad\" (UID: \"b3327f34-66f3-4a70-915a-aa5ad8685fad\") " Nov 03 14:40:17.286811 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.286771 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3327f34-66f3-4a70-915a-aa5ad8685fad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3327f34-66f3-4a70-915a-aa5ad8685fad-istio-data\") pod \"b3327f34-66f3-4a70-915a-aa5ad8685fad\" (UID: \"b3327f34-66f3-4a70-915a-aa5ad8685fad\") " Nov 03 14:40:17.287131 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.287085 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3327f34-66f3-4a70-915a-aa5ad8685fad-istio-token\" (UniqueName: \"kubernetes.io/projected/b3327f34-66f3-4a70-915a-aa5ad8685fad-istio-token\") pod \"b3327f34-66f3-4a70-915a-aa5ad8685fad\" (UID: \"b3327f34-66f3-4a70-915a-aa5ad8685fad\") " Nov 03 14:40:17.287327 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.287232 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3327f34-66f3-4a70-915a-aa5ad8685fad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3327f34-66f3-4a70-915a-aa5ad8685fad-workload-socket\") pod \"b3327f34-66f3-4a70-915a-aa5ad8685fad\" (UID: \"b3327f34-66f3-4a70-915a-aa5ad8685fad\") " Nov 03 14:40:17.287591 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.287556 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6695b0e-c9b6-4119-a0b2-508b212a3ece-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6695b0e-c9b6-4119-a0b2-508b212a3ece-workload-socket\") pod \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\" (UID: \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\") " Nov 03 14:40:17.287671 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.287623 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3327f34-66f3-4a70-915a-aa5ad8685fad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3327f34-66f3-4a70-915a-aa5ad8685fad-istiod-ca-cert\") pod \"b3327f34-66f3-4a70-915a-aa5ad8685fad\" (UID: \"b3327f34-66f3-4a70-915a-aa5ad8685fad\") " Nov 03 14:40:17.287671 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.287656 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6695b0e-c9b6-4119-a0b2-508b212a3ece-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6695b0e-c9b6-4119-a0b2-508b212a3ece-istiod-ca-cert\") pod \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\" (UID: \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\") " Nov 03 14:40:17.287789 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.287712 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6695b0e-c9b6-4119-a0b2-508b212a3ece-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6695b0e-c9b6-4119-a0b2-508b212a3ece-istio-data\") pod \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\" (UID: \"f6695b0e-c9b6-4119-a0b2-508b212a3ece\") " Nov 03 14:40:17.288399 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.288165 5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6695b0e-c9b6-4119-a0b2-508b212a3ece-credential-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.288399 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.288359 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6695b0e-c9b6-4119-a0b2-508b212a3ece-istio-envoy" pod "f6695b0e-c9b6-4119-a0b2-508b212a3ece" (UID: "f6695b0e-c9b6-4119-a0b2-508b212a3ece"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:40:17.288593 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.288459 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6695b0e-c9b6-4119-a0b2-508b212a3ece-kube-api-access-v4hq6" pod "f6695b0e-c9b6-4119-a0b2-508b212a3ece" (UID: "f6695b0e-c9b6-4119-a0b2-508b212a3ece"). InnerVolumeSpecName "kube-api-access-v4hq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:40:17.288593 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.288478 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6695b0e-c9b6-4119-a0b2-508b212a3ece-istio-data" pod "f6695b0e-c9b6-4119-a0b2-508b212a3ece" (UID: "f6695b0e-c9b6-4119-a0b2-508b212a3ece"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:40:17.288716 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.288674 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3327f34-66f3-4a70-915a-aa5ad8685fad-istio-envoy" pod "b3327f34-66f3-4a70-915a-aa5ad8685fad" (UID: "b3327f34-66f3-4a70-915a-aa5ad8685fad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:40:17.288781 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.288710 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6695b0e-c9b6-4119-a0b2-508b212a3ece-workload-socket" pod "f6695b0e-c9b6-4119-a0b2-508b212a3ece" (UID: "f6695b0e-c9b6-4119-a0b2-508b212a3ece"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:40:17.288781 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.288715 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3327f34-66f3-4a70-915a-aa5ad8685fad-kube-api-access-k5kq5" pod "b3327f34-66f3-4a70-915a-aa5ad8685fad" (UID: "b3327f34-66f3-4a70-915a-aa5ad8685fad"). InnerVolumeSpecName "kube-api-access-k5kq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:40:17.288781 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.288774 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3327f34-66f3-4a70-915a-aa5ad8685fad-istiod-ca-cert" pod "b3327f34-66f3-4a70-915a-aa5ad8685fad" (UID: "b3327f34-66f3-4a70-915a-aa5ad8685fad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:40:17.288984 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.288917 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3327f34-66f3-4a70-915a-aa5ad8685fad-workload-socket" pod "b3327f34-66f3-4a70-915a-aa5ad8685fad" (UID: "b3327f34-66f3-4a70-915a-aa5ad8685fad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:40:17.288984 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.288954 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6695b0e-c9b6-4119-a0b2-508b212a3ece-workload-certs" pod "f6695b0e-c9b6-4119-a0b2-508b212a3ece" (UID: "f6695b0e-c9b6-4119-a0b2-508b212a3ece"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:40:17.289099 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.289048 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6695b0e-c9b6-4119-a0b2-508b212a3ece-istiod-ca-cert" pod "f6695b0e-c9b6-4119-a0b2-508b212a3ece" (UID: "f6695b0e-c9b6-4119-a0b2-508b212a3ece"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:40:17.289174 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.289061 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6695b0e-c9b6-4119-a0b2-508b212a3ece-istio-token" pod "f6695b0e-c9b6-4119-a0b2-508b212a3ece" (UID: "f6695b0e-c9b6-4119-a0b2-508b212a3ece"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:40:17.289174 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.289084 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3327f34-66f3-4a70-915a-aa5ad8685fad-credential-socket" pod "b3327f34-66f3-4a70-915a-aa5ad8685fad" (UID: "b3327f34-66f3-4a70-915a-aa5ad8685fad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:40:17.289303 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.289118 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3327f34-66f3-4a70-915a-aa5ad8685fad-workload-certs" pod "b3327f34-66f3-4a70-915a-aa5ad8685fad" (UID: "b3327f34-66f3-4a70-915a-aa5ad8685fad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:40:17.289303 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.289275 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3327f34-66f3-4a70-915a-aa5ad8685fad-istio-data" pod "b3327f34-66f3-4a70-915a-aa5ad8685fad" (UID: "b3327f34-66f3-4a70-915a-aa5ad8685fad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:40:17.289786 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.289761 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6695b0e-c9b6-4119-a0b2-508b212a3ece-istio-podinfo" pod "f6695b0e-c9b6-4119-a0b2-508b212a3ece" (UID: "f6695b0e-c9b6-4119-a0b2-508b212a3ece"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:40:17.290343 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.290310 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3327f34-66f3-4a70-915a-aa5ad8685fad-istio-podinfo" pod "b3327f34-66f3-4a70-915a-aa5ad8685fad" (UID: "b3327f34-66f3-4a70-915a-aa5ad8685fad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:40:17.290960 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.290939 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3327f34-66f3-4a70-915a-aa5ad8685fad-istio-token" pod "b3327f34-66f3-4a70-915a-aa5ad8685fad" (UID: "b3327f34-66f3-4a70-915a-aa5ad8685fad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:40:17.388794 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.388719 5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6695b0e-c9b6-4119-a0b2-508b212a3ece-istio-envoy\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.388794 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.388750 5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3327f34-66f3-4a70-915a-aa5ad8685fad-istio-podinfo\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.388794 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.388760 5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6695b0e-c9b6-4119-a0b2-508b212a3ece-istio-token\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.388794 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.388769 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4hq6\" (UniqueName: \"kubernetes.io/projected/f6695b0e-c9b6-4119-a0b2-508b212a3ece-kube-api-access-v4hq6\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.388794 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.388780 5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3327f34-66f3-4a70-915a-aa5ad8685fad-workload-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.388794 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.388788 5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6695b0e-c9b6-4119-a0b2-508b212a3ece-workload-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.388794 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.388796 5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3327f34-66f3-4a70-915a-aa5ad8685fad-credential-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.389138 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.388803 5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3327f34-66f3-4a70-915a-aa5ad8685fad-istio-data\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.389138 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.388812 5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3327f34-66f3-4a70-915a-aa5ad8685fad-istio-token\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.389138 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.388820 5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3327f34-66f3-4a70-915a-aa5ad8685fad-workload-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.389138 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.388828 5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6695b0e-c9b6-4119-a0b2-508b212a3ece-workload-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.389138 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.388868 5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3327f34-66f3-4a70-915a-aa5ad8685fad-istiod-ca-cert\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.389138 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.388883 5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6695b0e-c9b6-4119-a0b2-508b212a3ece-istiod-ca-cert\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.389138 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.388894 5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6695b0e-c9b6-4119-a0b2-508b212a3ece-istio-data\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.389138 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.388906 5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6695b0e-c9b6-4119-a0b2-508b212a3ece-istio-podinfo\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.389138 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.388919 5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3327f34-66f3-4a70-915a-aa5ad8685fad-istio-envoy\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:40:17.389138 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:17.388927 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5kq5\" (UniqueName: \"kubernetes.io/projected/b3327f34-66f3-4a70-915a-aa5ad8685fad-kube-api-access-k5kq5\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:40:18.150676 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:18.150642 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" Nov 03 14:40:18.150676 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:18.150655 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz" event={"ID":"b3327f34-66f3-4a70-915a-aa5ad8685fad","Type":"ContainerDied","Data":"ee6708b33ac72e956689651504c8ee13d989bd5274bfdd3d9f37391e0b320047"} Nov 03 14:40:18.151225 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:18.150714 5060 scope.go:122] "RemoveContainer" containerID="42e4fcc0f5d87a398aeeabbebcddac52534e1612a7107b5f0f20c1d3b29b20a4" Nov 03 14:40:18.152412 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:18.152383 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" event={"ID":"f6695b0e-c9b6-4119-a0b2-508b212a3ece","Type":"ContainerDied","Data":"d2e3f15e556a5e9a39c5116cf348bb3e4079f3d353d4b4584a127dce278f5406"} Nov 03 14:40:18.152533 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:18.152393 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v" Nov 03 14:40:18.161136 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:18.161117 5060 scope.go:122] "RemoveContainer" containerID="b6f1d2902246456c639f4b454edd5fbb2ddab0a2d1c3474d794c7fdcb26b6673" Nov 03 14:40:18.223925 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:18.223811 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v"] Nov 03 14:40:18.236362 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:18.236333 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-2b94v"] Nov 03 14:40:18.249144 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:18.249117 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6695b0e-c9b6-4119-a0b2-508b212a3ece" path="/var/lib/kubelet/pods/f6695b0e-c9b6-4119-a0b2-508b212a3ece/volumes" Nov 03 14:40:18.263826 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:18.263803 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz"] Nov 03 14:40:18.266662 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:18.266639 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fkwwz"] Nov 03 14:40:20.249754 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:40:20.249720 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3327f34-66f3-4a70-915a-aa5ad8685fad" path="/var/lib/kubelet/pods/b3327f34-66f3-4a70-915a-aa5ad8685fad/volumes" Nov 03 14:40:39.439299 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:40:39.439258 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:41:07.418660 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:41:07.418632 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/3.log" Nov 03 14:41:07.424360 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:41:07.424333 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/3.log" Nov 03 14:41:09.439200 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:41:09.439158 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:41:39.439023 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:41:39.438983 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:42:09.439109 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:42:09.439065 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:42:38.042357 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:42:38.042321 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1af1f14-zgc6j"] Nov 03 14:42:38.054752 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:42:38.054722 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1af1f14-zgc6j"] Nov 03 14:42:38.248078 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:42:38.248045 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c29e0a5b-36c0-402f-90ca-b8c8fca88acd" path="/var/lib/kubelet/pods/c29e0a5b-36c0-402f-90ca-b8c8fca88acd/volumes" Nov 03 14:42:39.441296 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:42:39.441255 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:42:54.039646 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:42:54.039605 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9040ed96-zkclk"] Nov 03 14:42:54.052181 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:42:54.052149 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9040ed96-zkclk"] Nov 03 14:42:54.253490 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:42:54.253458 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cb8e7eb-3150-46e8-b9b6-48f53b265483" path="/var/lib/kubelet/pods/0cb8e7eb-3150-46e8-b9b6-48f53b265483/volumes" Nov 03 14:43:09.439052 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:43:09.439015 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:43:39.438977 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:43:39.438939 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:44:08.228371 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:44:08.228340 5060 scope.go:122] "RemoveContainer" containerID="3e8e3e65c8f572c1bd58abeae46e2b68afe56d34eb0d2a8aac8f8e9ec8a10f5c" Nov 03 14:44:08.242259 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:44:08.242234 5060 scope.go:122] "RemoveContainer" containerID="40110fad18a3c3df8e51cfd8c284bf4d3ca618cf47cfec86462906504ab203c1" Nov 03 14:44:09.440087 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:44:09.440049 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:44:28.038152 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:44:28.038118 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf4400e7-7bpsm"] Nov 03 14:44:28.050777 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:44:28.050751 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf4400e7-7bpsm"] Nov 03 14:44:28.248342 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:44:28.248308 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e634e8b3-2f74-482d-aaf5-4165f8294b45" path="/var/lib/kubelet/pods/e634e8b3-2f74-482d-aaf5-4165f8294b45/volumes" Nov 03 14:44:39.439417 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:44:39.439375 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:45:00.156799 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:00.156759 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895285-6hfzx"] Nov 03 14:45:00.157545 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:00.157523 5060 cpu_manager.go:414] "containerMap: removing container" podUID="59dcdef3-f88e-4463-a0dd-1917e915a516" containerName="main" Nov 03 14:45:00.157545 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:00.157548 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59dcdef3-f88e-4463-a0dd-1917e915a516" containerName="main" Nov 03 14:45:00.157735 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:00.157572 5060 cpu_manager.go:414] "containerMap: removing container" podUID="f6695b0e-c9b6-4119-a0b2-508b212a3ece" containerName="istio-proxy" Nov 03 14:45:00.157735 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:00.157579 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6695b0e-c9b6-4119-a0b2-508b212a3ece" containerName="istio-proxy" Nov 03 14:45:00.157735 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:00.157590 5060 cpu_manager.go:414] "containerMap: removing container" podUID="3aaaeea4-134f-4d17-a180-b4857c6425d3" containerName="main" Nov 03 14:45:00.157735 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:00.157597 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aaaeea4-134f-4d17-a180-b4857c6425d3" containerName="main" Nov 03 14:45:00.157735 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:00.157648 5060 cpu_manager.go:414] "containerMap: removing container" podUID="e634e8b3-2f74-482d-aaf5-4165f8294b45" containerName="revoke-keys" Nov 03 14:45:00.157735 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:00.157656 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e634e8b3-2f74-482d-aaf5-4165f8294b45" containerName="revoke-keys" Nov 03 14:45:00.157735 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:00.157669 5060 cpu_manager.go:414] "containerMap: removing container" podUID="b3327f34-66f3-4a70-915a-aa5ad8685fad" containerName="istio-proxy" Nov 03 14:45:00.157735 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:00.157679 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3327f34-66f3-4a70-915a-aa5ad8685fad" containerName="istio-proxy" Nov 03 14:45:00.157735 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:00.157692 5060 cpu_manager.go:414] "containerMap: removing container" podUID="a88d21ce-086a-414a-911a-5561e2cb8023" containerName="istio-proxy" Nov 03 14:45:00.157735 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:00.157701 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a88d21ce-086a-414a-911a-5561e2cb8023" containerName="istio-proxy" Nov 03 14:45:00.158262 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:00.157811 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="59dcdef3-f88e-4463-a0dd-1917e915a516" containerName="main" Nov 03 14:45:00.158262 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:00.157826 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c8d7e93-3520-4a18-916a-ac77ae09f885" containerName="istio-proxy" Nov 03 14:45:00.158262 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:00.157864 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6695b0e-c9b6-4119-a0b2-508b212a3ece" containerName="istio-proxy" Nov 03 14:45:00.158262 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:00.157874 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aaaeea4-134f-4d17-a180-b4857c6425d3" containerName="main" Nov 03 14:45:00.158262 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:00.157894 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e634e8b3-2f74-482d-aaf5-4165f8294b45" containerName="revoke-keys" Nov 03 14:45:00.158262 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:00.157903 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3327f34-66f3-4a70-915a-aa5ad8685fad" containerName="istio-proxy" Nov 03 14:45:00.158262 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:00.157915 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a88d21ce-086a-414a-911a-5561e2cb8023" containerName="istio-proxy" Nov 03 14:45:00.161459 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:00.161429 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895285-6hfzx" Nov 03 14:45:00.181134 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:00.181106 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:45:00.193413 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:00.193384 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:45:00.193694 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:00.193675 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\"" Nov 03 14:45:00.197017 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:00.196989 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895285-6hfzx"] Nov 03 14:45:00.203246 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:00.203223 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/570a0e0d-16ce-486b-a762-d1e2dee1711e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895285-6hfzx\" (UID: \"570a0e0d-16ce-486b-a762-d1e2dee1711e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895285-6hfzx" Nov 03 14:45:00.203426 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:00.203401 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbxc8\" (UniqueName: \"kubernetes.io/projected/570a0e0d-16ce-486b-a762-d1e2dee1711e-kube-api-access-qbxc8\") pod \"maas-api-key-cleanup-29895285-6hfzx\" (UID: \"570a0e0d-16ce-486b-a762-d1e2dee1711e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895285-6hfzx" Nov 03 14:45:00.304599 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:00.304553 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbxc8\" (UniqueName: \"kubernetes.io/projected/570a0e0d-16ce-486b-a762-d1e2dee1711e-kube-api-access-qbxc8\") pod \"maas-api-key-cleanup-29895285-6hfzx\" (UID: \"570a0e0d-16ce-486b-a762-d1e2dee1711e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895285-6hfzx" Nov 03 14:45:00.304785 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:00.304664 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/570a0e0d-16ce-486b-a762-d1e2dee1711e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895285-6hfzx\" (UID: \"570a0e0d-16ce-486b-a762-d1e2dee1711e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895285-6hfzx" Nov 03 14:45:00.305494 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:00.305466 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/570a0e0d-16ce-486b-a762-d1e2dee1711e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895285-6hfzx\" (UID: \"570a0e0d-16ce-486b-a762-d1e2dee1711e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895285-6hfzx" Nov 03 14:45:00.327210 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:00.327181 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbxc8\" (UniqueName: \"kubernetes.io/projected/570a0e0d-16ce-486b-a762-d1e2dee1711e-kube-api-access-qbxc8\") pod \"maas-api-key-cleanup-29895285-6hfzx\" (UID: \"570a0e0d-16ce-486b-a762-d1e2dee1711e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895285-6hfzx" Nov 03 14:45:00.474931 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:00.474814 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895285-6hfzx" Nov 03 14:45:00.780159 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:00.780084 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895285-6hfzx"] Nov 03 14:45:00.782849 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:45:00.782799 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod570a0e0d_16ce_486b_a762_d1e2dee1711e.slice/crio-771ed94237793102b620dc48e7c24dd64370f0ab9e145fd619e553fff6a622d3 WatchSource:0}: Error finding container 771ed94237793102b620dc48e7c24dd64370f0ab9e145fd619e553fff6a622d3: Status 404 returned error can't find the container with id 771ed94237793102b620dc48e7c24dd64370f0ab9e145fd619e553fff6a622d3 Nov 03 14:45:00.812090 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:00.812064 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895285-6hfzx" event={"ID":"570a0e0d-16ce-486b-a762-d1e2dee1711e","Type":"ContainerStarted","Data":"771ed94237793102b620dc48e7c24dd64370f0ab9e145fd619e553fff6a622d3"} Nov 03 14:45:01.626666 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:45:01.626635 5060 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod570a0e0d_16ce_486b_a762_d1e2dee1711e.slice/crio-cbfca8dc04f75e529be7fc1bba3aa89a6bec2421a68d39443d8dde902ac22147.scope\": RecentStats: unable to find data in memory cache]" Nov 03 14:45:01.819757 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:01.819723 5060 generic.go:356] "Generic (PLEG): container finished" podID="570a0e0d-16ce-486b-a762-d1e2dee1711e" containerID="cbfca8dc04f75e529be7fc1bba3aa89a6bec2421a68d39443d8dde902ac22147" exitCode=60 Nov 03 14:45:01.819951 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:01.819810 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895285-6hfzx" event={"ID":"570a0e0d-16ce-486b-a762-d1e2dee1711e","Type":"ContainerDied","Data":"cbfca8dc04f75e529be7fc1bba3aa89a6bec2421a68d39443d8dde902ac22147"} Nov 03 14:45:01.820027 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:01.820012 5060 scope.go:122] "RemoveContainer" containerID="cbfca8dc04f75e529be7fc1bba3aa89a6bec2421a68d39443d8dde902ac22147" Nov 03 14:45:02.828190 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:02.828155 5060 generic.go:356] "Generic (PLEG): container finished" podID="570a0e0d-16ce-486b-a762-d1e2dee1711e" containerID="ad8ea9931bb8af8bea15cbfeaa06e3e93364d0fb7202370bc34b8dca28bed0b2" exitCode=60 Nov 03 14:45:02.828718 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:02.828212 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895285-6hfzx" event={"ID":"570a0e0d-16ce-486b-a762-d1e2dee1711e","Type":"ContainerDied","Data":"ad8ea9931bb8af8bea15cbfeaa06e3e93364d0fb7202370bc34b8dca28bed0b2"} Nov 03 14:45:02.828718 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:02.828266 5060 scope.go:122] "RemoveContainer" containerID="cbfca8dc04f75e529be7fc1bba3aa89a6bec2421a68d39443d8dde902ac22147" Nov 03 14:45:02.828718 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:02.828574 5060 scope.go:122] "RemoveContainer" containerID="ad8ea9931bb8af8bea15cbfeaa06e3e93364d0fb7202370bc34b8dca28bed0b2" Nov 03 14:45:02.828931 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:45:02.828906 5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895285-6hfzx_odh-ai-gateway-infra(570a0e0d-16ce-486b-a762-d1e2dee1711e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895285-6hfzx" podUID="570a0e0d-16ce-486b-a762-d1e2dee1711e" Nov 03 14:45:03.835433 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:03.835407 5060 scope.go:122] "RemoveContainer" containerID="ad8ea9931bb8af8bea15cbfeaa06e3e93364d0fb7202370bc34b8dca28bed0b2" Nov 03 14:45:03.835845 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:45:03.835641 5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895285-6hfzx_odh-ai-gateway-infra(570a0e0d-16ce-486b-a762-d1e2dee1711e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895285-6hfzx" podUID="570a0e0d-16ce-486b-a762-d1e2dee1711e" Nov 03 14:45:09.440114 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:45:09.440075 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:45:14.242650 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:14.242618 5060 scope.go:122] "RemoveContainer" containerID="ad8ea9931bb8af8bea15cbfeaa06e3e93364d0fb7202370bc34b8dca28bed0b2" Nov 03 14:45:14.899664 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:14.899581 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895285-6hfzx" event={"ID":"570a0e0d-16ce-486b-a762-d1e2dee1711e","Type":"ContainerStarted","Data":"25a08149f494bdf77992561f30d24de8ce18c2bc6fd8ea4d67d81c64a6e0eec9"} Nov 03 14:45:14.945895 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:14.945847 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895285-6hfzx" podStartSLOduration=14.945818351 podStartE2EDuration="14.945818351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:45:14.943393052 +0000 UTC m=+2048.832182633" watchObservedRunningTime="2026-11-03 14:45:14.945818351 +0000 UTC m=+2048.834607933" Nov 03 14:45:15.907129 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:15.907093 5060 generic.go:356] "Generic (PLEG): container finished" podID="570a0e0d-16ce-486b-a762-d1e2dee1711e" containerID="25a08149f494bdf77992561f30d24de8ce18c2bc6fd8ea4d67d81c64a6e0eec9" exitCode=60 Nov 03 14:45:15.907616 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:15.907167 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895285-6hfzx" event={"ID":"570a0e0d-16ce-486b-a762-d1e2dee1711e","Type":"ContainerDied","Data":"25a08149f494bdf77992561f30d24de8ce18c2bc6fd8ea4d67d81c64a6e0eec9"} Nov 03 14:45:15.907616 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:15.907245 5060 scope.go:122] "RemoveContainer" containerID="ad8ea9931bb8af8bea15cbfeaa06e3e93364d0fb7202370bc34b8dca28bed0b2" Nov 03 14:45:15.907616 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:15.907521 5060 scope.go:122] "RemoveContainer" containerID="25a08149f494bdf77992561f30d24de8ce18c2bc6fd8ea4d67d81c64a6e0eec9" Nov 03 14:45:15.907791 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:45:15.907770 5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895285-6hfzx_odh-ai-gateway-infra(570a0e0d-16ce-486b-a762-d1e2dee1711e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895285-6hfzx" podUID="570a0e0d-16ce-486b-a762-d1e2dee1711e" Nov 03 14:45:15.959355 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:15.959328 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895285-6hfzx"] Nov 03 14:45:17.049712 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:17.049687 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895285-6hfzx" Nov 03 14:45:17.176694 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:17.176611 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/570a0e0d-16ce-486b-a762-d1e2dee1711e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/570a0e0d-16ce-486b-a762-d1e2dee1711e-maas-api-service-ca\") pod \"570a0e0d-16ce-486b-a762-d1e2dee1711e\" (UID: \"570a0e0d-16ce-486b-a762-d1e2dee1711e\") " Nov 03 14:45:17.176694 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:17.176667 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/570a0e0d-16ce-486b-a762-d1e2dee1711e-kube-api-access-qbxc8\" (UniqueName: \"kubernetes.io/projected/570a0e0d-16ce-486b-a762-d1e2dee1711e-kube-api-access-qbxc8\") pod \"570a0e0d-16ce-486b-a762-d1e2dee1711e\" (UID: \"570a0e0d-16ce-486b-a762-d1e2dee1711e\") " Nov 03 14:45:17.176958 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:17.176938 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/570a0e0d-16ce-486b-a762-d1e2dee1711e-maas-api-service-ca" pod "570a0e0d-16ce-486b-a762-d1e2dee1711e" (UID: "570a0e0d-16ce-486b-a762-d1e2dee1711e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:45:17.178714 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:17.178688 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/570a0e0d-16ce-486b-a762-d1e2dee1711e-kube-api-access-qbxc8" pod "570a0e0d-16ce-486b-a762-d1e2dee1711e" (UID: "570a0e0d-16ce-486b-a762-d1e2dee1711e"). InnerVolumeSpecName "kube-api-access-qbxc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:45:17.277959 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:17.277930 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbxc8\" (UniqueName: \"kubernetes.io/projected/570a0e0d-16ce-486b-a762-d1e2dee1711e-kube-api-access-qbxc8\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:17.277959 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:17.277956 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/570a0e0d-16ce-486b-a762-d1e2dee1711e-maas-api-service-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:17.919748 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:17.919717 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895285-6hfzx" Nov 03 14:45:17.919928 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:17.919742 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895285-6hfzx" event={"ID":"570a0e0d-16ce-486b-a762-d1e2dee1711e","Type":"ContainerDied","Data":"771ed94237793102b620dc48e7c24dd64370f0ab9e145fd619e553fff6a622d3"} Nov 03 14:45:17.919928 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:17.919804 5060 scope.go:122] "RemoveContainer" containerID="25a08149f494bdf77992561f30d24de8ce18c2bc6fd8ea4d67d81c64a6e0eec9" Nov 03 14:45:17.977766 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:17.977742 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895285-6hfzx"] Nov 03 14:45:17.985457 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:17.985430 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895285-6hfzx"] Nov 03 14:45:18.255252 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:18.255219 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="570a0e0d-16ce-486b-a762-d1e2dee1711e" path="/var/lib/kubelet/pods/570a0e0d-16ce-486b-a762-d1e2dee1711e/volumes" Nov 03 14:45:31.690467 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:31.690432 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt"] Nov 03 14:45:31.691014 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:31.690642 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" podUID="c0653ff5-b9b7-4ed9-8311-df75f12109ca" containerName="istio-proxy" containerID="cri-o://f7043cf8764bec87135a994490e9f26507a7fb93c88374f2e16b6cb3745a2585" gracePeriod=30 Nov 03 14:45:36.940235 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:36.940208 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" Nov 03 14:45:37.029773 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:37.029739 5060 generic.go:356] "Generic (PLEG): container finished" podID="c0653ff5-b9b7-4ed9-8311-df75f12109ca" containerID="f7043cf8764bec87135a994490e9f26507a7fb93c88374f2e16b6cb3745a2585" exitCode=0 Nov 03 14:45:37.029995 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:37.029799 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" Nov 03 14:45:37.029995 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:37.029820 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" event={"ID":"c0653ff5-b9b7-4ed9-8311-df75f12109ca","Type":"ContainerDied","Data":"f7043cf8764bec87135a994490e9f26507a7fb93c88374f2e16b6cb3745a2585"} Nov 03 14:45:37.029995 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:37.029892 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt" event={"ID":"c0653ff5-b9b7-4ed9-8311-df75f12109ca","Type":"ContainerDied","Data":"75036d10805cf2fb507efcf916e8e896b1dec096c0dbb5e4300eee0dfbd7e047"} Nov 03 14:45:37.029995 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:37.029916 5060 scope.go:122] "RemoveContainer" containerID="f7043cf8764bec87135a994490e9f26507a7fb93c88374f2e16b6cb3745a2585" Nov 03 14:45:37.039037 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:37.039019 5060 scope.go:122] "RemoveContainer" containerID="f7043cf8764bec87135a994490e9f26507a7fb93c88374f2e16b6cb3745a2585" Nov 03 14:45:37.039325 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:45:37.039301 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7043cf8764bec87135a994490e9f26507a7fb93c88374f2e16b6cb3745a2585\": container with ID starting with f7043cf8764bec87135a994490e9f26507a7fb93c88374f2e16b6cb3745a2585 not found: ID does not exist" containerID="f7043cf8764bec87135a994490e9f26507a7fb93c88374f2e16b6cb3745a2585" Nov 03 14:45:37.039401 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:37.039338 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7043cf8764bec87135a994490e9f26507a7fb93c88374f2e16b6cb3745a2585"} err="failed to get container status \"f7043cf8764bec87135a994490e9f26507a7fb93c88374f2e16b6cb3745a2585\": rpc error: code = NotFound desc = could not find container \"f7043cf8764bec87135a994490e9f26507a7fb93c88374f2e16b6cb3745a2585\": container with ID starting with f7043cf8764bec87135a994490e9f26507a7fb93c88374f2e16b6cb3745a2585 not found: ID does not exist" Nov 03 14:45:37.077263 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:37.077233 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0653ff5-b9b7-4ed9-8311-df75f12109ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0653ff5-b9b7-4ed9-8311-df75f12109ca-credential-socket\") pod \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\" (UID: \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\") " Nov 03 14:45:37.077406 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:37.077270 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0653ff5-b9b7-4ed9-8311-df75f12109ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0653ff5-b9b7-4ed9-8311-df75f12109ca-workload-certs\") pod \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\" (UID: \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\") " Nov 03 14:45:37.077406 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:37.077339 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0653ff5-b9b7-4ed9-8311-df75f12109ca-kube-api-access-nfs2l\" (UniqueName: \"kubernetes.io/projected/c0653ff5-b9b7-4ed9-8311-df75f12109ca-kube-api-access-nfs2l\") pod \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\" (UID: \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\") " Nov 03 14:45:37.077406 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:37.077377 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0653ff5-b9b7-4ed9-8311-df75f12109ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0653ff5-b9b7-4ed9-8311-df75f12109ca-workload-socket\") pod \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\" (UID: \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\") " Nov 03 14:45:37.077577 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:37.077421 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0653ff5-b9b7-4ed9-8311-df75f12109ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0653ff5-b9b7-4ed9-8311-df75f12109ca-istiod-ca-cert\") pod \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\" (UID: \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\") " Nov 03 14:45:37.077577 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:37.077471 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0653ff5-b9b7-4ed9-8311-df75f12109ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0653ff5-b9b7-4ed9-8311-df75f12109ca-istio-data\") pod \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\" (UID: \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\") " Nov 03 14:45:37.077577 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:37.077505 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0653ff5-b9b7-4ed9-8311-df75f12109ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0653ff5-b9b7-4ed9-8311-df75f12109ca-istio-envoy\") pod \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\" (UID: \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\") " Nov 03 14:45:37.077577 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:37.077506 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0653ff5-b9b7-4ed9-8311-df75f12109ca-credential-socket" pod "c0653ff5-b9b7-4ed9-8311-df75f12109ca" (UID: "c0653ff5-b9b7-4ed9-8311-df75f12109ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:37.077867 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:37.077603 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0653ff5-b9b7-4ed9-8311-df75f12109ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0653ff5-b9b7-4ed9-8311-df75f12109ca-istio-podinfo\") pod \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\" (UID: \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\") " Nov 03 14:45:37.077867 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:37.077644 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0653ff5-b9b7-4ed9-8311-df75f12109ca-workload-certs" pod "c0653ff5-b9b7-4ed9-8311-df75f12109ca" (UID: "c0653ff5-b9b7-4ed9-8311-df75f12109ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:37.077867 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:37.077692 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0653ff5-b9b7-4ed9-8311-df75f12109ca-istio-token\" (UniqueName: \"kubernetes.io/projected/c0653ff5-b9b7-4ed9-8311-df75f12109ca-istio-token\") pod \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\" (UID: \"c0653ff5-b9b7-4ed9-8311-df75f12109ca\") " Nov 03 14:45:37.077867 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:37.077705 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0653ff5-b9b7-4ed9-8311-df75f12109ca-workload-socket" pod "c0653ff5-b9b7-4ed9-8311-df75f12109ca" (UID: "c0653ff5-b9b7-4ed9-8311-df75f12109ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:37.078087 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:37.077905 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0653ff5-b9b7-4ed9-8311-df75f12109ca-istio-data" pod "c0653ff5-b9b7-4ed9-8311-df75f12109ca" (UID: "c0653ff5-b9b7-4ed9-8311-df75f12109ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:37.078087 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:37.077917 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0653ff5-b9b7-4ed9-8311-df75f12109ca-istiod-ca-cert" pod "c0653ff5-b9b7-4ed9-8311-df75f12109ca" (UID: "c0653ff5-b9b7-4ed9-8311-df75f12109ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:45:37.078215 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:37.078102 5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0653ff5-b9b7-4ed9-8311-df75f12109ca-credential-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:37.078215 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:37.078124 5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0653ff5-b9b7-4ed9-8311-df75f12109ca-workload-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:37.078215 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:37.078139 5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0653ff5-b9b7-4ed9-8311-df75f12109ca-workload-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:37.078215 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:37.078154 5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0653ff5-b9b7-4ed9-8311-df75f12109ca-istiod-ca-cert\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:37.078215 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:37.078168 5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0653ff5-b9b7-4ed9-8311-df75f12109ca-istio-data\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:37.079916 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:37.079882 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0653ff5-b9b7-4ed9-8311-df75f12109ca-kube-api-access-nfs2l" pod "c0653ff5-b9b7-4ed9-8311-df75f12109ca" (UID: "c0653ff5-b9b7-4ed9-8311-df75f12109ca"). InnerVolumeSpecName "kube-api-access-nfs2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:45:37.080014 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:37.079963 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0653ff5-b9b7-4ed9-8311-df75f12109ca-istio-podinfo" pod "c0653ff5-b9b7-4ed9-8311-df75f12109ca" (UID: "c0653ff5-b9b7-4ed9-8311-df75f12109ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:45:37.080147 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:37.080127 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0653ff5-b9b7-4ed9-8311-df75f12109ca-istio-envoy" pod "c0653ff5-b9b7-4ed9-8311-df75f12109ca" (UID: "c0653ff5-b9b7-4ed9-8311-df75f12109ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:37.080211 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:37.080164 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0653ff5-b9b7-4ed9-8311-df75f12109ca-istio-token" pod "c0653ff5-b9b7-4ed9-8311-df75f12109ca" (UID: "c0653ff5-b9b7-4ed9-8311-df75f12109ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:45:37.179145 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:37.179114 5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0653ff5-b9b7-4ed9-8311-df75f12109ca-istio-token\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:37.179145 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:37.179145 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfs2l\" (UniqueName: \"kubernetes.io/projected/c0653ff5-b9b7-4ed9-8311-df75f12109ca-kube-api-access-nfs2l\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:37.179354 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:37.179160 5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0653ff5-b9b7-4ed9-8311-df75f12109ca-istio-envoy\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:37.179354 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:37.179174 5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0653ff5-b9b7-4ed9-8311-df75f12109ca-istio-podinfo\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:37.378988 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:37.378953 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt"] Nov 03 14:45:37.391767 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:37.391738 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-cgtqt"] Nov 03 14:45:38.248370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:38.248336 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0653ff5-b9b7-4ed9-8311-df75f12109ca" path="/var/lib/kubelet/pods/c0653ff5-b9b7-4ed9-8311-df75f12109ca/volumes" Nov 03 14:45:39.438786 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:45:39.438748 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:45:48.920040 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:48.920000 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2"] Nov 03 14:45:48.920660 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:48.920287 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" podUID="941df1f9-9c29-4205-b2fb-88167bd809ec" containerName="istio-proxy" containerID="cri-o://7fda49450467c3bae83dc94c20f58447d0c2bc9916da8a2ce29e541ed4c2637b" gracePeriod=30 Nov 03 14:45:48.932532 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:48.932507 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4"] Nov 03 14:45:48.932726 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:48.932692 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" podUID="d073cee6-107f-4a7f-8117-f48de055e723" containerName="istio-proxy" containerID="cri-o://9fca417cc6621f31361585bf48ee6da880ad1cb71a7c8a62e2112b4cb6f61e41" gracePeriod=30 Nov 03 14:45:49.054402 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:49.054365 5060 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 14:45:49.054611 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:49.054430 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" podUID="d073cee6-107f-4a7f-8117-f48de055e723" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 14:45:50.237797 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:50.237761 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-jf2qh_37e492af-6e43-4224-be07-6e92f69bbae1/manager/0.log" Nov 03 14:45:50.449975 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:50.449936 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-t7s86_57e263ee-fe23-475e-aa0d-72f40f837809/manager/0.log" Nov 03 14:45:51.348173 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:51.348102 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-82pkj_b0d2f12e-9c6c-4e22-ba6f-197f411b0666/manager/0.log" Nov 03 14:45:51.761636 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:51.761601 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-569f5_3ced846c-60f0-4206-836e-f94c281d6438/manager/0.log" Nov 03 14:45:52.888167 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:52.888139 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-4ggmd_7b8c342f-20c2-406f-a6a1-f8eef4118a6d/manager/0.log" Nov 03 14:45:53.286301 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:53.286268 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-54d5w_218bc458-4d44-453a-84c8-2b7fe211580e/manager/0.log" Nov 03 14:45:53.292575 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:53.292548 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x"] Nov 03 14:45:53.292757 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:53.292734 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" podUID="50a3ab6a-5b07-4d38-b00b-9b77b506ec20" containerName="istio-proxy" containerID="cri-o://5a01c8b6ee5b43f71a8bea2356448d9f9f738e75225baffa9712cc50e7b596bd" gracePeriod=30 Nov 03 14:45:53.305541 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:53.305511 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z"] Nov 03 14:45:53.305707 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:53.305685 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" podUID="2d7d5e9c-625a-42af-98d7-52b9a8b5745e" containerName="istio-proxy" containerID="cri-o://bbc9a0ada347660eae0ce19b39a6050c28f4884c260bf3b04884871bab49c9d2" gracePeriod=30 Nov 03 14:45:53.307588 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:53.307566 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg"] Nov 03 14:45:53.307780 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:53.307759 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" podUID="9737ed5d-be9c-45ef-afd6-977eddcc3ce6" containerName="istio-proxy" containerID="cri-o://6268b68cb5af006ac4996e17d774cc0536c0536bcdfd1bfc5eb67ed7fc2f46a6" gracePeriod=30 Nov 03 14:45:53.324951 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:53.324911 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7"] Nov 03 14:45:53.325253 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:53.325206 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" podUID="43465f0f-764a-44c5-b762-9b6932953b41" containerName="istio-proxy" containerID="cri-o://2eebb02b02b15cd18301087ba05ebb49a583321cc83f4dd05a702c9591ad8226" gracePeriod=30 Nov 03 14:45:54.143476 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.143444 5060 generic.go:356] "Generic (PLEG): container finished" podID="941df1f9-9c29-4205-b2fb-88167bd809ec" containerID="7fda49450467c3bae83dc94c20f58447d0c2bc9916da8a2ce29e541ed4c2637b" exitCode=0 Nov 03 14:45:54.143959 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.143549 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" event={"ID":"941df1f9-9c29-4205-b2fb-88167bd809ec","Type":"ContainerDied","Data":"7fda49450467c3bae83dc94c20f58447d0c2bc9916da8a2ce29e541ed4c2637b"} Nov 03 14:45:54.145633 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.145598 5060 generic.go:356] "Generic (PLEG): container finished" podID="d073cee6-107f-4a7f-8117-f48de055e723" containerID="9fca417cc6621f31361585bf48ee6da880ad1cb71a7c8a62e2112b4cb6f61e41" exitCode=0 Nov 03 14:45:54.145767 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.145663 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" event={"ID":"d073cee6-107f-4a7f-8117-f48de055e723","Type":"ContainerDied","Data":"9fca417cc6621f31361585bf48ee6da880ad1cb71a7c8a62e2112b4cb6f61e41"} Nov 03 14:45:54.145767 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.145710 5060 scope.go:122] "RemoveContainer" containerID="3a2d2a056be578877ce5697216806387880856c9389bcbb2718b214e15251ab6" Nov 03 14:45:54.210591 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.210534 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" Nov 03 14:45:54.226895 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.226875 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" Nov 03 14:45:54.340824 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.340778 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d073cee6-107f-4a7f-8117-f48de055e723-kube-api-access-5ql9q\" (UniqueName: \"kubernetes.io/projected/d073cee6-107f-4a7f-8117-f48de055e723-kube-api-access-5ql9q\") pod \"d073cee6-107f-4a7f-8117-f48de055e723\" (UID: \"d073cee6-107f-4a7f-8117-f48de055e723\") " Nov 03 14:45:54.340824 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.340825 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/941df1f9-9c29-4205-b2fb-88167bd809ec-kube-api-access-nzbj9\" (UniqueName: \"kubernetes.io/projected/941df1f9-9c29-4205-b2fb-88167bd809ec-kube-api-access-nzbj9\") pod \"941df1f9-9c29-4205-b2fb-88167bd809ec\" (UID: \"941df1f9-9c29-4205-b2fb-88167bd809ec\") " Nov 03 14:45:54.341108 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.340889 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/941df1f9-9c29-4205-b2fb-88167bd809ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/941df1f9-9c29-4205-b2fb-88167bd809ec-istio-envoy\") pod \"941df1f9-9c29-4205-b2fb-88167bd809ec\" (UID: \"941df1f9-9c29-4205-b2fb-88167bd809ec\") " Nov 03 14:45:54.341108 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.340936 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/941df1f9-9c29-4205-b2fb-88167bd809ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/941df1f9-9c29-4205-b2fb-88167bd809ec-credential-socket\") pod \"941df1f9-9c29-4205-b2fb-88167bd809ec\" (UID: \"941df1f9-9c29-4205-b2fb-88167bd809ec\") " Nov 03 14:45:54.341108 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.340981 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d073cee6-107f-4a7f-8117-f48de055e723-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d073cee6-107f-4a7f-8117-f48de055e723-istiod-ca-cert\") pod \"d073cee6-107f-4a7f-8117-f48de055e723\" (UID: \"d073cee6-107f-4a7f-8117-f48de055e723\") " Nov 03 14:45:54.341108 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.341021 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d073cee6-107f-4a7f-8117-f48de055e723-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d073cee6-107f-4a7f-8117-f48de055e723-istio-podinfo\") pod \"d073cee6-107f-4a7f-8117-f48de055e723\" (UID: \"d073cee6-107f-4a7f-8117-f48de055e723\") " Nov 03 14:45:54.341108 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.341051 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/941df1f9-9c29-4205-b2fb-88167bd809ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/941df1f9-9c29-4205-b2fb-88167bd809ec-istio-podinfo\") pod \"941df1f9-9c29-4205-b2fb-88167bd809ec\" (UID: \"941df1f9-9c29-4205-b2fb-88167bd809ec\") " Nov 03 14:45:54.341108 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.341070 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/941df1f9-9c29-4205-b2fb-88167bd809ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/941df1f9-9c29-4205-b2fb-88167bd809ec-istio-data\") pod \"941df1f9-9c29-4205-b2fb-88167bd809ec\" (UID: \"941df1f9-9c29-4205-b2fb-88167bd809ec\") " Nov 03 14:45:54.341108 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.341105 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/941df1f9-9c29-4205-b2fb-88167bd809ec-istio-token\" (UniqueName: \"kubernetes.io/projected/941df1f9-9c29-4205-b2fb-88167bd809ec-istio-token\") pod \"941df1f9-9c29-4205-b2fb-88167bd809ec\" (UID: \"941df1f9-9c29-4205-b2fb-88167bd809ec\") " Nov 03 14:45:54.341494 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.341134 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d073cee6-107f-4a7f-8117-f48de055e723-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d073cee6-107f-4a7f-8117-f48de055e723-istio-data\") pod \"d073cee6-107f-4a7f-8117-f48de055e723\" (UID: \"d073cee6-107f-4a7f-8117-f48de055e723\") " Nov 03 14:45:54.341494 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.341165 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d073cee6-107f-4a7f-8117-f48de055e723-istio-token\" (UniqueName: \"kubernetes.io/projected/d073cee6-107f-4a7f-8117-f48de055e723-istio-token\") pod \"d073cee6-107f-4a7f-8117-f48de055e723\" (UID: \"d073cee6-107f-4a7f-8117-f48de055e723\") " Nov 03 14:45:54.341494 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.341194 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d073cee6-107f-4a7f-8117-f48de055e723-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d073cee6-107f-4a7f-8117-f48de055e723-workload-socket\") pod \"d073cee6-107f-4a7f-8117-f48de055e723\" (UID: \"d073cee6-107f-4a7f-8117-f48de055e723\") " Nov 03 14:45:54.341494 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.341224 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/941df1f9-9c29-4205-b2fb-88167bd809ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/941df1f9-9c29-4205-b2fb-88167bd809ec-workload-socket\") pod \"941df1f9-9c29-4205-b2fb-88167bd809ec\" (UID: \"941df1f9-9c29-4205-b2fb-88167bd809ec\") " Nov 03 14:45:54.341494 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.341258 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d073cee6-107f-4a7f-8117-f48de055e723-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d073cee6-107f-4a7f-8117-f48de055e723-credential-socket\") pod \"d073cee6-107f-4a7f-8117-f48de055e723\" (UID: \"d073cee6-107f-4a7f-8117-f48de055e723\") " Nov 03 14:45:54.341494 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.341296 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/941df1f9-9c29-4205-b2fb-88167bd809ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/941df1f9-9c29-4205-b2fb-88167bd809ec-workload-certs\") pod \"941df1f9-9c29-4205-b2fb-88167bd809ec\" (UID: \"941df1f9-9c29-4205-b2fb-88167bd809ec\") " Nov 03 14:45:54.341494 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.341333 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d073cee6-107f-4a7f-8117-f48de055e723-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d073cee6-107f-4a7f-8117-f48de055e723-workload-certs\") pod \"d073cee6-107f-4a7f-8117-f48de055e723\" (UID: \"d073cee6-107f-4a7f-8117-f48de055e723\") " Nov 03 14:45:54.341494 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.341365 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/941df1f9-9c29-4205-b2fb-88167bd809ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/941df1f9-9c29-4205-b2fb-88167bd809ec-istiod-ca-cert\") pod \"941df1f9-9c29-4205-b2fb-88167bd809ec\" (UID: \"941df1f9-9c29-4205-b2fb-88167bd809ec\") " Nov 03 14:45:54.341494 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.341418 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d073cee6-107f-4a7f-8117-f48de055e723-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d073cee6-107f-4a7f-8117-f48de055e723-istio-envoy\") pod \"d073cee6-107f-4a7f-8117-f48de055e723\" (UID: \"d073cee6-107f-4a7f-8117-f48de055e723\") " Nov 03 14:45:54.341984 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.341747 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/941df1f9-9c29-4205-b2fb-88167bd809ec-credential-socket" pod "941df1f9-9c29-4205-b2fb-88167bd809ec" (UID: "941df1f9-9c29-4205-b2fb-88167bd809ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:54.342064 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.342039 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/941df1f9-9c29-4205-b2fb-88167bd809ec-workload-certs" pod "941df1f9-9c29-4205-b2fb-88167bd809ec" (UID: "941df1f9-9c29-4205-b2fb-88167bd809ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:54.342128 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.342106 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d073cee6-107f-4a7f-8117-f48de055e723-credential-socket" pod "d073cee6-107f-4a7f-8117-f48de055e723" (UID: "d073cee6-107f-4a7f-8117-f48de055e723"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:54.342272 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.342250 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d073cee6-107f-4a7f-8117-f48de055e723-workload-certs" pod "d073cee6-107f-4a7f-8117-f48de055e723" (UID: "d073cee6-107f-4a7f-8117-f48de055e723"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:54.342352 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.342329 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d073cee6-107f-4a7f-8117-f48de055e723-istio-data" pod "d073cee6-107f-4a7f-8117-f48de055e723" (UID: "d073cee6-107f-4a7f-8117-f48de055e723"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:54.342576 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.342551 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/941df1f9-9c29-4205-b2fb-88167bd809ec-istiod-ca-cert" pod "941df1f9-9c29-4205-b2fb-88167bd809ec" (UID: "941df1f9-9c29-4205-b2fb-88167bd809ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:45:54.343620 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.343585 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/941df1f9-9c29-4205-b2fb-88167bd809ec-workload-socket" pod "941df1f9-9c29-4205-b2fb-88167bd809ec" (UID: "941df1f9-9c29-4205-b2fb-88167bd809ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:54.343889 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.343657 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/941df1f9-9c29-4205-b2fb-88167bd809ec-istio-data" pod "941df1f9-9c29-4205-b2fb-88167bd809ec" (UID: "941df1f9-9c29-4205-b2fb-88167bd809ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:54.343889 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.343847 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d073cee6-107f-4a7f-8117-f48de055e723-workload-socket" pod "d073cee6-107f-4a7f-8117-f48de055e723" (UID: "d073cee6-107f-4a7f-8117-f48de055e723"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:54.344174 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.343890 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d073cee6-107f-4a7f-8117-f48de055e723-istiod-ca-cert" pod "d073cee6-107f-4a7f-8117-f48de055e723" (UID: "d073cee6-107f-4a7f-8117-f48de055e723"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:45:54.344680 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.344520 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/941df1f9-9c29-4205-b2fb-88167bd809ec-istio-token" pod "941df1f9-9c29-4205-b2fb-88167bd809ec" (UID: "941df1f9-9c29-4205-b2fb-88167bd809ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:45:54.344680 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.344520 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/941df1f9-9c29-4205-b2fb-88167bd809ec-istio-envoy" pod "941df1f9-9c29-4205-b2fb-88167bd809ec" (UID: "941df1f9-9c29-4205-b2fb-88167bd809ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:54.344680 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.344646 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d073cee6-107f-4a7f-8117-f48de055e723-kube-api-access-5ql9q" pod "d073cee6-107f-4a7f-8117-f48de055e723" (UID: "d073cee6-107f-4a7f-8117-f48de055e723"). InnerVolumeSpecName "kube-api-access-5ql9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:45:54.345031 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.345008 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d073cee6-107f-4a7f-8117-f48de055e723-istio-token" pod "d073cee6-107f-4a7f-8117-f48de055e723" (UID: "d073cee6-107f-4a7f-8117-f48de055e723"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:45:54.345139 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.345038 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d073cee6-107f-4a7f-8117-f48de055e723-istio-envoy" pod "d073cee6-107f-4a7f-8117-f48de055e723" (UID: "d073cee6-107f-4a7f-8117-f48de055e723"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:54.345884 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.345828 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/941df1f9-9c29-4205-b2fb-88167bd809ec-kube-api-access-nzbj9" pod "941df1f9-9c29-4205-b2fb-88167bd809ec" (UID: "941df1f9-9c29-4205-b2fb-88167bd809ec"). InnerVolumeSpecName "kube-api-access-nzbj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:45:54.346292 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.346261 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d073cee6-107f-4a7f-8117-f48de055e723-istio-podinfo" pod "d073cee6-107f-4a7f-8117-f48de055e723" (UID: "d073cee6-107f-4a7f-8117-f48de055e723"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:45:54.346396 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.346350 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/941df1f9-9c29-4205-b2fb-88167bd809ec-istio-podinfo" pod "941df1f9-9c29-4205-b2fb-88167bd809ec" (UID: "941df1f9-9c29-4205-b2fb-88167bd809ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:45:54.442978 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.442945 5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d073cee6-107f-4a7f-8117-f48de055e723-istio-token\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:54.442978 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.442973 5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d073cee6-107f-4a7f-8117-f48de055e723-workload-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:54.442978 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.442983 5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/941df1f9-9c29-4205-b2fb-88167bd809ec-workload-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:54.443223 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.442992 5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d073cee6-107f-4a7f-8117-f48de055e723-credential-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:54.443223 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.443002 5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/941df1f9-9c29-4205-b2fb-88167bd809ec-workload-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:54.443223 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.443012 5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d073cee6-107f-4a7f-8117-f48de055e723-workload-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:54.443223 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.443020 5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/941df1f9-9c29-4205-b2fb-88167bd809ec-istiod-ca-cert\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:54.443223 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.443028 5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d073cee6-107f-4a7f-8117-f48de055e723-istio-envoy\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:54.443223 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.443036 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ql9q\" (UniqueName: \"kubernetes.io/projected/d073cee6-107f-4a7f-8117-f48de055e723-kube-api-access-5ql9q\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:54.443223 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.443045 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzbj9\" (UniqueName: \"kubernetes.io/projected/941df1f9-9c29-4205-b2fb-88167bd809ec-kube-api-access-nzbj9\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:54.443223 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.443053 5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/941df1f9-9c29-4205-b2fb-88167bd809ec-istio-envoy\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:54.443223 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.443061 5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/941df1f9-9c29-4205-b2fb-88167bd809ec-credential-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:54.443223 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.443069 5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d073cee6-107f-4a7f-8117-f48de055e723-istiod-ca-cert\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:54.443223 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.443076 5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d073cee6-107f-4a7f-8117-f48de055e723-istio-podinfo\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:54.443223 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.443085 5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/941df1f9-9c29-4205-b2fb-88167bd809ec-istio-podinfo\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:54.443223 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.443092 5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/941df1f9-9c29-4205-b2fb-88167bd809ec-istio-data\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:54.443223 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.443101 5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/941df1f9-9c29-4205-b2fb-88167bd809ec-istio-token\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:54.443223 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:54.443108 5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d073cee6-107f-4a7f-8117-f48de055e723-istio-data\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:55.153285 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:55.153253 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" Nov 03 14:45:55.153740 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:55.153252 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2" event={"ID":"941df1f9-9c29-4205-b2fb-88167bd809ec","Type":"ContainerDied","Data":"e30c2177fbbbaf259ea24af6e4f7166a2702985d8d072a1f18ebabc9c2b0cc3d"} Nov 03 14:45:55.153740 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:55.153398 5060 scope.go:122] "RemoveContainer" containerID="7fda49450467c3bae83dc94c20f58447d0c2bc9916da8a2ce29e541ed4c2637b" Nov 03 14:45:55.154797 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:55.154772 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" event={"ID":"d073cee6-107f-4a7f-8117-f48de055e723","Type":"ContainerDied","Data":"c824f12ba8115293aa70ac08fb835a423cd38b6fadb7c80228938eca97faac56"} Nov 03 14:45:55.154797 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:55.154788 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4" Nov 03 14:45:55.162636 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:55.162618 5060 scope.go:122] "RemoveContainer" containerID="9fca417cc6621f31361585bf48ee6da880ad1cb71a7c8a62e2112b4cb6f61e41" Nov 03 14:45:55.237212 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:55.237142 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4"] Nov 03 14:45:55.250136 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:55.250103 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-qx4p4"] Nov 03 14:45:55.293851 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:55.293813 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2"] Nov 03 14:45:55.303775 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:55.303745 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-bttp2"] Nov 03 14:45:56.248893 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:56.248853 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="941df1f9-9c29-4205-b2fb-88167bd809ec" path="/var/lib/kubelet/pods/941df1f9-9c29-4205-b2fb-88167bd809ec/volumes" Nov 03 14:45:56.249505 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:56.249482 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d073cee6-107f-4a7f-8117-f48de055e723" path="/var/lib/kubelet/pods/d073cee6-107f-4a7f-8117-f48de055e723/volumes" Nov 03 14:45:58.568478 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.568455 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" Nov 03 14:45:58.618003 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.617966 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" Nov 03 14:45:58.653800 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.653764 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" Nov 03 14:45:58.661672 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.661648 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" Nov 03 14:45:58.682112 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.682075 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-kube-api-access-pdtlx\" (UniqueName: \"kubernetes.io/projected/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-kube-api-access-pdtlx\") pod \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\" (UID: \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\") " Nov 03 14:45:58.682262 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.682130 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43465f0f-764a-44c5-b762-9b6932953b41-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43465f0f-764a-44c5-b762-9b6932953b41-workload-certs\") pod \"43465f0f-764a-44c5-b762-9b6932953b41\" (UID: \"43465f0f-764a-44c5-b762-9b6932953b41\") " Nov 03 14:45:58.682262 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.682184 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-workload-certs\") pod \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\" (UID: \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\") " Nov 03 14:45:58.682262 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.682214 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43465f0f-764a-44c5-b762-9b6932953b41-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43465f0f-764a-44c5-b762-9b6932953b41-istio-data\") pod \"43465f0f-764a-44c5-b762-9b6932953b41\" (UID: \"43465f0f-764a-44c5-b762-9b6932953b41\") " Nov 03 14:45:58.682262 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.682257 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-istio-podinfo\") pod \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\" (UID: \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\") " Nov 03 14:45:58.682489 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.682300 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-workload-socket\") pod \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\" (UID: \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\") " Nov 03 14:45:58.682489 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.682332 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43465f0f-764a-44c5-b762-9b6932953b41-istio-token\" (UniqueName: \"kubernetes.io/projected/43465f0f-764a-44c5-b762-9b6932953b41-istio-token\") pod \"43465f0f-764a-44c5-b762-9b6932953b41\" (UID: \"43465f0f-764a-44c5-b762-9b6932953b41\") " Nov 03 14:45:58.682489 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.682370 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43465f0f-764a-44c5-b762-9b6932953b41-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43465f0f-764a-44c5-b762-9b6932953b41-istio-podinfo\") pod \"43465f0f-764a-44c5-b762-9b6932953b41\" (UID: \"43465f0f-764a-44c5-b762-9b6932953b41\") " Nov 03 14:45:58.682489 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.682401 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43465f0f-764a-44c5-b762-9b6932953b41-kube-api-access-jjcsn\" (UniqueName: \"kubernetes.io/projected/43465f0f-764a-44c5-b762-9b6932953b41-kube-api-access-jjcsn\") pod \"43465f0f-764a-44c5-b762-9b6932953b41\" (UID: \"43465f0f-764a-44c5-b762-9b6932953b41\") " Nov 03 14:45:58.682489 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.682427 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-istio-token\" (UniqueName: \"kubernetes.io/projected/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-istio-token\") pod \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\" (UID: \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\") " Nov 03 14:45:58.682489 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.682461 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-istiod-ca-cert\") pod \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\" (UID: \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\") " Nov 03 14:45:58.682801 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.682500 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43465f0f-764a-44c5-b762-9b6932953b41-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43465f0f-764a-44c5-b762-9b6932953b41-workload-socket\") pod \"43465f0f-764a-44c5-b762-9b6932953b41\" (UID: \"43465f0f-764a-44c5-b762-9b6932953b41\") " Nov 03 14:45:58.682801 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.682530 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-istio-envoy\") pod \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\" (UID: \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\") " Nov 03 14:45:58.682801 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.682580 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43465f0f-764a-44c5-b762-9b6932953b41-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43465f0f-764a-44c5-b762-9b6932953b41-credential-socket\") pod \"43465f0f-764a-44c5-b762-9b6932953b41\" (UID: \"43465f0f-764a-44c5-b762-9b6932953b41\") " Nov 03 14:45:58.682801 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.682621 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-istio-data\") pod \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\" (UID: \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\") " Nov 03 14:45:58.682801 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.682659 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43465f0f-764a-44c5-b762-9b6932953b41-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43465f0f-764a-44c5-b762-9b6932953b41-istiod-ca-cert\") pod \"43465f0f-764a-44c5-b762-9b6932953b41\" (UID: \"43465f0f-764a-44c5-b762-9b6932953b41\") " Nov 03 14:45:58.682801 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.682693 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-credential-socket\") pod \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\" (UID: \"9737ed5d-be9c-45ef-afd6-977eddcc3ce6\") " Nov 03 14:45:58.682801 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.682751 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43465f0f-764a-44c5-b762-9b6932953b41-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43465f0f-764a-44c5-b762-9b6932953b41-istio-envoy\") pod \"43465f0f-764a-44c5-b762-9b6932953b41\" (UID: \"43465f0f-764a-44c5-b762-9b6932953b41\") " Nov 03 14:45:58.684064 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.682423 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43465f0f-764a-44c5-b762-9b6932953b41-workload-certs" pod "43465f0f-764a-44c5-b762-9b6932953b41" (UID: "43465f0f-764a-44c5-b762-9b6932953b41"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:58.684064 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.682644 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-workload-socket" pod "9737ed5d-be9c-45ef-afd6-977eddcc3ce6" (UID: "9737ed5d-be9c-45ef-afd6-977eddcc3ce6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:58.685153 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.684069 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43465f0f-764a-44c5-b762-9b6932953b41-workload-socket" pod "43465f0f-764a-44c5-b762-9b6932953b41" (UID: "43465f0f-764a-44c5-b762-9b6932953b41"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:58.685153 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.682942 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-istiod-ca-cert" pod "9737ed5d-be9c-45ef-afd6-977eddcc3ce6" (UID: "9737ed5d-be9c-45ef-afd6-977eddcc3ce6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:45:58.685153 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.683033 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43465f0f-764a-44c5-b762-9b6932953b41-istio-data" pod "43465f0f-764a-44c5-b762-9b6932953b41" (UID: "43465f0f-764a-44c5-b762-9b6932953b41"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:58.685153 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.683655 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-workload-certs" pod "9737ed5d-be9c-45ef-afd6-977eddcc3ce6" (UID: "9737ed5d-be9c-45ef-afd6-977eddcc3ce6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:58.685153 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.683793 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43465f0f-764a-44c5-b762-9b6932953b41-credential-socket" pod "43465f0f-764a-44c5-b762-9b6932953b41" (UID: "43465f0f-764a-44c5-b762-9b6932953b41"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:58.685153 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.684805 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-kube-api-access-pdtlx" pod "9737ed5d-be9c-45ef-afd6-977eddcc3ce6" (UID: "9737ed5d-be9c-45ef-afd6-977eddcc3ce6"). InnerVolumeSpecName "kube-api-access-pdtlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:45:58.685153 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.684904 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43465f0f-764a-44c5-b762-9b6932953b41-istiod-ca-cert" pod "43465f0f-764a-44c5-b762-9b6932953b41" (UID: "43465f0f-764a-44c5-b762-9b6932953b41"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:45:58.685153 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.684971 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-istio-data" pod "9737ed5d-be9c-45ef-afd6-977eddcc3ce6" (UID: "9737ed5d-be9c-45ef-afd6-977eddcc3ce6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:58.686339 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.685380 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-credential-socket" pod "9737ed5d-be9c-45ef-afd6-977eddcc3ce6" (UID: "9737ed5d-be9c-45ef-afd6-977eddcc3ce6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:58.687063 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.687038 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-istio-envoy" pod "9737ed5d-be9c-45ef-afd6-977eddcc3ce6" (UID: "9737ed5d-be9c-45ef-afd6-977eddcc3ce6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:58.687151 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.687082 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43465f0f-764a-44c5-b762-9b6932953b41-kube-api-access-jjcsn" pod "43465f0f-764a-44c5-b762-9b6932953b41" (UID: "43465f0f-764a-44c5-b762-9b6932953b41"). InnerVolumeSpecName "kube-api-access-jjcsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:45:58.687151 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.687117 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43465f0f-764a-44c5-b762-9b6932953b41-istio-envoy" pod "43465f0f-764a-44c5-b762-9b6932953b41" (UID: "43465f0f-764a-44c5-b762-9b6932953b41"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:58.687232 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.687177 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43465f0f-764a-44c5-b762-9b6932953b41-istio-token" pod "43465f0f-764a-44c5-b762-9b6932953b41" (UID: "43465f0f-764a-44c5-b762-9b6932953b41"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:45:58.687372 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.687353 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-istio-podinfo" pod "9737ed5d-be9c-45ef-afd6-977eddcc3ce6" (UID: "9737ed5d-be9c-45ef-afd6-977eddcc3ce6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:45:58.688152 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.688135 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43465f0f-764a-44c5-b762-9b6932953b41-istio-podinfo" pod "43465f0f-764a-44c5-b762-9b6932953b41" (UID: "43465f0f-764a-44c5-b762-9b6932953b41"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:45:58.688740 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.688717 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-istio-token" pod "9737ed5d-be9c-45ef-afd6-977eddcc3ce6" (UID: "9737ed5d-be9c-45ef-afd6-977eddcc3ce6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:45:58.783753 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.783721 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-istiod-ca-cert\") pod \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\" (UID: \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\") " Nov 03 14:45:58.783935 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.783769 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-istio-token\" (UniqueName: \"kubernetes.io/projected/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-istio-token\") pod \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\" (UID: \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\") " Nov 03 14:45:58.783935 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.783803 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-istio-podinfo\") pod \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\" (UID: \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\") " Nov 03 14:45:58.783935 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.783859 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-workload-socket\") pod \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\" (UID: \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\") " Nov 03 14:45:58.783935 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.783890 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-workload-certs\") pod \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\" (UID: \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\") " Nov 03 14:45:58.783935 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.783933 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-workload-certs\") pod \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\" (UID: \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\") " Nov 03 14:45:58.784220 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.783963 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-credential-socket\") pod \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\" (UID: \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\") " Nov 03 14:45:58.784220 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.783990 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-credential-socket\") pod \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\" (UID: \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\") " Nov 03 14:45:58.784220 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.784034 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-kube-api-access-q2b4r\" (UniqueName: \"kubernetes.io/projected/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-kube-api-access-q2b4r\") pod \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\" (UID: \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\") " Nov 03 14:45:58.784220 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.784061 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-istio-data\") pod \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\" (UID: \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\") " Nov 03 14:45:58.784220 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.784098 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-istio-data\") pod \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\" (UID: \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\") " Nov 03 14:45:58.784220 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.784131 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-kube-api-access-rl8sp\" (UniqueName: \"kubernetes.io/projected/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-kube-api-access-rl8sp\") pod \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\" (UID: \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\") " Nov 03 14:45:58.784220 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.784178 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-istio-envoy\") pod \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\" (UID: \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\") " Nov 03 14:45:58.784220 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.784209 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-istiod-ca-cert\") pod \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\" (UID: \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\") " Nov 03 14:45:58.784630 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.784250 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-istio-podinfo\") pod \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\" (UID: \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\") " Nov 03 14:45:58.784630 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.784255 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-credential-socket" pod "50a3ab6a-5b07-4d38-b00b-9b77b506ec20" (UID: "50a3ab6a-5b07-4d38-b00b-9b77b506ec20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:58.784630 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.784283 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-istio-token\" (UniqueName: \"kubernetes.io/projected/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-istio-token\") pod \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\" (UID: \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\") " Nov 03 14:45:58.784630 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.784260 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-workload-certs" pod "2d7d5e9c-625a-42af-98d7-52b9a8b5745e" (UID: "2d7d5e9c-625a-42af-98d7-52b9a8b5745e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:58.784630 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.784290 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-istiod-ca-cert" pod "2d7d5e9c-625a-42af-98d7-52b9a8b5745e" (UID: "2d7d5e9c-625a-42af-98d7-52b9a8b5745e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:45:58.784630 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.784327 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-workload-socket\") pod \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\" (UID: \"2d7d5e9c-625a-42af-98d7-52b9a8b5745e\") " Nov 03 14:45:58.784630 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.784387 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-istio-envoy\") pod \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\" (UID: \"50a3ab6a-5b07-4d38-b00b-9b77b506ec20\") " Nov 03 14:45:58.784630 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.784394 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-workload-socket" pod "50a3ab6a-5b07-4d38-b00b-9b77b506ec20" (UID: "50a3ab6a-5b07-4d38-b00b-9b77b506ec20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:58.784630 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.784494 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-istio-data" pod "2d7d5e9c-625a-42af-98d7-52b9a8b5745e" (UID: "2d7d5e9c-625a-42af-98d7-52b9a8b5745e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:58.784630 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.784625 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-istio-data" pod "50a3ab6a-5b07-4d38-b00b-9b77b506ec20" (UID: "50a3ab6a-5b07-4d38-b00b-9b77b506ec20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:58.785172 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.784635 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-workload-certs" pod "50a3ab6a-5b07-4d38-b00b-9b77b506ec20" (UID: "50a3ab6a-5b07-4d38-b00b-9b77b506ec20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:58.785172 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.784755 5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43465f0f-764a-44c5-b762-9b6932953b41-istio-token\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.785172 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.784802 5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43465f0f-764a-44c5-b762-9b6932953b41-istio-podinfo\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.785172 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.784817 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjcsn\" (UniqueName: \"kubernetes.io/projected/43465f0f-764a-44c5-b762-9b6932953b41-kube-api-access-jjcsn\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.785172 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.784849 5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-istio-token\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.785172 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.784865 5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-istiod-ca-cert\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.785172 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.784878 5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-istiod-ca-cert\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.785172 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.784894 5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43465f0f-764a-44c5-b762-9b6932953b41-workload-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.785172 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.784908 5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-istio-envoy\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.785172 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.784923 5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43465f0f-764a-44c5-b762-9b6932953b41-credential-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.785172 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.784936 5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-istio-data\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.785172 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.784951 5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43465f0f-764a-44c5-b762-9b6932953b41-istiod-ca-cert\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.785172 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.784964 5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-credential-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.785172 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.784977 5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-workload-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.785172 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.784991 5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-workload-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.785172 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.785005 5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43465f0f-764a-44c5-b762-9b6932953b41-istio-envoy\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.785172 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.785020 5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-workload-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.785172 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.785033 5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-credential-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.785172 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.785049 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdtlx\" (UniqueName: \"kubernetes.io/projected/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-kube-api-access-pdtlx\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.785172 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.785063 5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-istio-data\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.785172 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.785076 5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-istio-data\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.785172 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.785089 5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43465f0f-764a-44c5-b762-9b6932953b41-workload-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.785172 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.785106 5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-workload-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.785172 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.785125 5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43465f0f-764a-44c5-b762-9b6932953b41-istio-data\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.785172 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.785138 5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-istio-podinfo\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.785172 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.785151 5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9737ed5d-be9c-45ef-afd6-977eddcc3ce6-workload-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.785172 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.784849 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-credential-socket" pod "2d7d5e9c-625a-42af-98d7-52b9a8b5745e" (UID: "2d7d5e9c-625a-42af-98d7-52b9a8b5745e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:58.786635 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.785602 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-istiod-ca-cert" pod "50a3ab6a-5b07-4d38-b00b-9b77b506ec20" (UID: "50a3ab6a-5b07-4d38-b00b-9b77b506ec20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:45:58.786635 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.786179 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-workload-socket" pod "2d7d5e9c-625a-42af-98d7-52b9a8b5745e" (UID: "2d7d5e9c-625a-42af-98d7-52b9a8b5745e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:58.786635 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.786591 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-istio-token" pod "2d7d5e9c-625a-42af-98d7-52b9a8b5745e" (UID: "2d7d5e9c-625a-42af-98d7-52b9a8b5745e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:45:58.786823 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.786801 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-istio-envoy" pod "50a3ab6a-5b07-4d38-b00b-9b77b506ec20" (UID: "50a3ab6a-5b07-4d38-b00b-9b77b506ec20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:58.786899 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.786805 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-istio-podinfo" pod "50a3ab6a-5b07-4d38-b00b-9b77b506ec20" (UID: "50a3ab6a-5b07-4d38-b00b-9b77b506ec20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:45:58.787192 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.787160 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-kube-api-access-rl8sp" pod "2d7d5e9c-625a-42af-98d7-52b9a8b5745e" (UID: "2d7d5e9c-625a-42af-98d7-52b9a8b5745e"). InnerVolumeSpecName "kube-api-access-rl8sp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:45:58.787289 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.787165 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-istio-envoy" pod "2d7d5e9c-625a-42af-98d7-52b9a8b5745e" (UID: "2d7d5e9c-625a-42af-98d7-52b9a8b5745e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:45:58.787289 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.787233 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-istio-podinfo" pod "2d7d5e9c-625a-42af-98d7-52b9a8b5745e" (UID: "2d7d5e9c-625a-42af-98d7-52b9a8b5745e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:45:58.788334 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.788317 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-kube-api-access-q2b4r" pod "50a3ab6a-5b07-4d38-b00b-9b77b506ec20" (UID: "50a3ab6a-5b07-4d38-b00b-9b77b506ec20"). InnerVolumeSpecName "kube-api-access-q2b4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:45:58.788503 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.788486 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-istio-token" pod "50a3ab6a-5b07-4d38-b00b-9b77b506ec20" (UID: "50a3ab6a-5b07-4d38-b00b-9b77b506ec20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:45:58.886033 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.885959 5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-istio-token\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.886033 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.885984 5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-istio-podinfo\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.886033 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.885995 5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-credential-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.886033 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.886006 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2b4r\" (UniqueName: \"kubernetes.io/projected/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-kube-api-access-q2b4r\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.886033 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.886015 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl8sp\" (UniqueName: \"kubernetes.io/projected/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-kube-api-access-rl8sp\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.886033 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.886025 5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-istio-envoy\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.886033 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.886035 5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-istiod-ca-cert\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.886357 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.886045 5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-istio-podinfo\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.886357 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.886053 5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-istio-token\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.886357 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.886061 5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7d5e9c-625a-42af-98d7-52b9a8b5745e-workload-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:58.886357 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:58.886070 5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50a3ab6a-5b07-4d38-b00b-9b77b506ec20-istio-envoy\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:45:59.186752 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:59.186659 5060 generic.go:356] "Generic (PLEG): container finished" podID="50a3ab6a-5b07-4d38-b00b-9b77b506ec20" containerID="5a01c8b6ee5b43f71a8bea2356448d9f9f738e75225baffa9712cc50e7b596bd" exitCode=0 Nov 03 14:45:59.186963 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:59.186762 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" Nov 03 14:45:59.186963 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:59.186794 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" event={"ID":"50a3ab6a-5b07-4d38-b00b-9b77b506ec20","Type":"ContainerDied","Data":"5a01c8b6ee5b43f71a8bea2356448d9f9f738e75225baffa9712cc50e7b596bd"} Nov 03 14:45:59.186963 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:59.186885 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x" event={"ID":"50a3ab6a-5b07-4d38-b00b-9b77b506ec20","Type":"ContainerDied","Data":"ab8dd4cd68812eface9e2ebd48b614f5a3255311e8be3ec879eff6e721653630"} Nov 03 14:45:59.186963 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:59.186916 5060 scope.go:122] "RemoveContainer" containerID="5a01c8b6ee5b43f71a8bea2356448d9f9f738e75225baffa9712cc50e7b596bd" Nov 03 14:45:59.188476 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:59.188450 5060 generic.go:356] "Generic (PLEG): container finished" podID="2d7d5e9c-625a-42af-98d7-52b9a8b5745e" containerID="bbc9a0ada347660eae0ce19b39a6050c28f4884c260bf3b04884871bab49c9d2" exitCode=0 Nov 03 14:45:59.188631 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:59.188498 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" Nov 03 14:45:59.188631 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:59.188545 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" event={"ID":"2d7d5e9c-625a-42af-98d7-52b9a8b5745e","Type":"ContainerDied","Data":"bbc9a0ada347660eae0ce19b39a6050c28f4884c260bf3b04884871bab49c9d2"} Nov 03 14:45:59.188631 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:59.188606 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z" event={"ID":"2d7d5e9c-625a-42af-98d7-52b9a8b5745e","Type":"ContainerDied","Data":"07ee22ae1936865181729d4520c439765f167be06ce2bc9075acf98411ad6022"} Nov 03 14:45:59.190072 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:59.190051 5060 generic.go:356] "Generic (PLEG): container finished" podID="9737ed5d-be9c-45ef-afd6-977eddcc3ce6" containerID="6268b68cb5af006ac4996e17d774cc0536c0536bcdfd1bfc5eb67ed7fc2f46a6" exitCode=0 Nov 03 14:45:59.190203 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:59.190118 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" Nov 03 14:45:59.190203 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:59.190144 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" event={"ID":"9737ed5d-be9c-45ef-afd6-977eddcc3ce6","Type":"ContainerDied","Data":"6268b68cb5af006ac4996e17d774cc0536c0536bcdfd1bfc5eb67ed7fc2f46a6"} Nov 03 14:45:59.190203 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:59.190181 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" event={"ID":"9737ed5d-be9c-45ef-afd6-977eddcc3ce6","Type":"ContainerDied","Data":"9a4550eb023faf820f9f50bba0a68f1cc15bdb402e465bc0459df2c1afa64913"} Nov 03 14:45:59.200074 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:59.200054 5060 generic.go:356] "Generic (PLEG): container finished" podID="43465f0f-764a-44c5-b762-9b6932953b41" containerID="2eebb02b02b15cd18301087ba05ebb49a583321cc83f4dd05a702c9591ad8226" exitCode=0 Nov 03 14:45:59.200170 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:59.200151 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" event={"ID":"43465f0f-764a-44c5-b762-9b6932953b41","Type":"ContainerDied","Data":"2eebb02b02b15cd18301087ba05ebb49a583321cc83f4dd05a702c9591ad8226"} Nov 03 14:45:59.200244 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:59.200193 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" event={"ID":"43465f0f-764a-44c5-b762-9b6932953b41","Type":"ContainerDied","Data":"0cb0aacb1086e7062da8a6481f750797cf3443a9abe14a9b70011b78d8f1ff14"} Nov 03 14:45:59.200390 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:59.200257 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7" Nov 03 14:45:59.200390 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:59.200326 5060 scope.go:122] "RemoveContainer" containerID="5a01c8b6ee5b43f71a8bea2356448d9f9f738e75225baffa9712cc50e7b596bd" Nov 03 14:45:59.200612 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:45:59.200595 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a01c8b6ee5b43f71a8bea2356448d9f9f738e75225baffa9712cc50e7b596bd\": container with ID starting with 5a01c8b6ee5b43f71a8bea2356448d9f9f738e75225baffa9712cc50e7b596bd not found: ID does not exist" containerID="5a01c8b6ee5b43f71a8bea2356448d9f9f738e75225baffa9712cc50e7b596bd" Nov 03 14:45:59.200672 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:59.200624 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a01c8b6ee5b43f71a8bea2356448d9f9f738e75225baffa9712cc50e7b596bd"} err="failed to get container status \"5a01c8b6ee5b43f71a8bea2356448d9f9f738e75225baffa9712cc50e7b596bd\": rpc error: code = NotFound desc = could not find container \"5a01c8b6ee5b43f71a8bea2356448d9f9f738e75225baffa9712cc50e7b596bd\": container with ID starting with 5a01c8b6ee5b43f71a8bea2356448d9f9f738e75225baffa9712cc50e7b596bd not found: ID does not exist" Nov 03 14:45:59.200672 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:59.200650 5060 scope.go:122] "RemoveContainer" containerID="bbc9a0ada347660eae0ce19b39a6050c28f4884c260bf3b04884871bab49c9d2" Nov 03 14:45:59.210769 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:59.210747 5060 scope.go:122] "RemoveContainer" containerID="bbc9a0ada347660eae0ce19b39a6050c28f4884c260bf3b04884871bab49c9d2" Nov 03 14:45:59.211089 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:45:59.211059 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbc9a0ada347660eae0ce19b39a6050c28f4884c260bf3b04884871bab49c9d2\": container with ID starting with bbc9a0ada347660eae0ce19b39a6050c28f4884c260bf3b04884871bab49c9d2 not found: ID does not exist" containerID="bbc9a0ada347660eae0ce19b39a6050c28f4884c260bf3b04884871bab49c9d2" Nov 03 14:45:59.211198 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:59.211094 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbc9a0ada347660eae0ce19b39a6050c28f4884c260bf3b04884871bab49c9d2"} err="failed to get container status \"bbc9a0ada347660eae0ce19b39a6050c28f4884c260bf3b04884871bab49c9d2\": rpc error: code = NotFound desc = could not find container \"bbc9a0ada347660eae0ce19b39a6050c28f4884c260bf3b04884871bab49c9d2\": container with ID starting with bbc9a0ada347660eae0ce19b39a6050c28f4884c260bf3b04884871bab49c9d2 not found: ID does not exist" Nov 03 14:45:59.211198 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:59.211110 5060 scope.go:122] "RemoveContainer" containerID="6268b68cb5af006ac4996e17d774cc0536c0536bcdfd1bfc5eb67ed7fc2f46a6" Nov 03 14:45:59.219475 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:59.219459 5060 scope.go:122] "RemoveContainer" containerID="6268b68cb5af006ac4996e17d774cc0536c0536bcdfd1bfc5eb67ed7fc2f46a6" Nov 03 14:45:59.219720 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:45:59.219701 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6268b68cb5af006ac4996e17d774cc0536c0536bcdfd1bfc5eb67ed7fc2f46a6\": container with ID starting with 6268b68cb5af006ac4996e17d774cc0536c0536bcdfd1bfc5eb67ed7fc2f46a6 not found: ID does not exist" containerID="6268b68cb5af006ac4996e17d774cc0536c0536bcdfd1bfc5eb67ed7fc2f46a6" Nov 03 14:45:59.219775 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:59.219726 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6268b68cb5af006ac4996e17d774cc0536c0536bcdfd1bfc5eb67ed7fc2f46a6"} err="failed to get container status \"6268b68cb5af006ac4996e17d774cc0536c0536bcdfd1bfc5eb67ed7fc2f46a6\": rpc error: code = NotFound desc = could not find container \"6268b68cb5af006ac4996e17d774cc0536c0536bcdfd1bfc5eb67ed7fc2f46a6\": container with ID starting with 6268b68cb5af006ac4996e17d774cc0536c0536bcdfd1bfc5eb67ed7fc2f46a6 not found: ID does not exist" Nov 03 14:45:59.219775 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:59.219742 5060 scope.go:122] "RemoveContainer" containerID="2eebb02b02b15cd18301087ba05ebb49a583321cc83f4dd05a702c9591ad8226" Nov 03 14:45:59.229122 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:59.229103 5060 scope.go:122] "RemoveContainer" containerID="2eebb02b02b15cd18301087ba05ebb49a583321cc83f4dd05a702c9591ad8226" Nov 03 14:45:59.229383 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:45:59.229362 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eebb02b02b15cd18301087ba05ebb49a583321cc83f4dd05a702c9591ad8226\": container with ID starting with 2eebb02b02b15cd18301087ba05ebb49a583321cc83f4dd05a702c9591ad8226 not found: ID does not exist" containerID="2eebb02b02b15cd18301087ba05ebb49a583321cc83f4dd05a702c9591ad8226" Nov 03 14:45:59.229449 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:59.229391 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eebb02b02b15cd18301087ba05ebb49a583321cc83f4dd05a702c9591ad8226"} err="failed to get container status \"2eebb02b02b15cd18301087ba05ebb49a583321cc83f4dd05a702c9591ad8226\": rpc error: code = NotFound desc = could not find container \"2eebb02b02b15cd18301087ba05ebb49a583321cc83f4dd05a702c9591ad8226\": container with ID starting with 2eebb02b02b15cd18301087ba05ebb49a583321cc83f4dd05a702c9591ad8226 not found: ID does not exist" Nov 03 14:45:59.300820 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:59.300781 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z"] Nov 03 14:45:59.309639 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:59.309608 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wcq9z"] Nov 03 14:45:59.350454 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:59.350413 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7"] Nov 03 14:45:59.352955 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:59.352930 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5h7r7"] Nov 03 14:45:59.401075 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:59.401044 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x"] Nov 03 14:45:59.416727 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:59.416699 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-rr46x"] Nov 03 14:45:59.462041 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:59.461966 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg"] Nov 03 14:45:59.466520 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:59.466494 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg"] Nov 03 14:45:59.562559 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:59.562526 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:45:59.562701 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:45:59.562580 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bzkwg" podUID="9737ed5d-be9c-45ef-afd6-977eddcc3ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:46:00.248392 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:46:00.248357 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d7d5e9c-625a-42af-98d7-52b9a8b5745e" path="/var/lib/kubelet/pods/2d7d5e9c-625a-42af-98d7-52b9a8b5745e/volumes" Nov 03 14:46:00.248991 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:46:00.248971 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43465f0f-764a-44c5-b762-9b6932953b41" path="/var/lib/kubelet/pods/43465f0f-764a-44c5-b762-9b6932953b41/volumes" Nov 03 14:46:00.249453 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:46:00.249434 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50a3ab6a-5b07-4d38-b00b-9b77b506ec20" path="/var/lib/kubelet/pods/50a3ab6a-5b07-4d38-b00b-9b77b506ec20/volumes" Nov 03 14:46:00.250208 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:46:00.250184 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9737ed5d-be9c-45ef-afd6-977eddcc3ce6" path="/var/lib/kubelet/pods/9737ed5d-be9c-45ef-afd6-977eddcc3ce6/volumes" Nov 03 14:46:07.456584 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:46:07.456556 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/3.log" Nov 03 14:46:07.458261 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:46:07.458221 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/3.log" Nov 03 14:46:08.278002 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:46:08.277971 5060 scope.go:122] "RemoveContainer" containerID="51c1decf449affacb7440685a1ef6d4efa2e33b66ad244ca702fc45a34062ac6" Nov 03 14:46:09.438443 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:46:09.438405 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:46:39.438971 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:46:39.438933 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:47:08.304131 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.304095 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9"] Nov 03 14:47:08.304675 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.304473 5060 cpu_manager.go:414] "containerMap: removing container" podUID="570a0e0d-16ce-486b-a762-d1e2dee1711e" containerName="cleanup" Nov 03 14:47:08.304675 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.304489 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="570a0e0d-16ce-486b-a762-d1e2dee1711e" containerName="cleanup" Nov 03 14:47:08.304675 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.304502 5060 cpu_manager.go:414] "containerMap: removing container" podUID="570a0e0d-16ce-486b-a762-d1e2dee1711e" containerName="cleanup" Nov 03 14:47:08.304675 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.304510 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="570a0e0d-16ce-486b-a762-d1e2dee1711e" containerName="cleanup" Nov 03 14:47:08.304675 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.304526 5060 cpu_manager.go:414] "containerMap: removing container" podUID="570a0e0d-16ce-486b-a762-d1e2dee1711e" containerName="cleanup" Nov 03 14:47:08.304675 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.304531 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="570a0e0d-16ce-486b-a762-d1e2dee1711e" containerName="cleanup" Nov 03 14:47:08.304675 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.304538 5060 cpu_manager.go:414] "containerMap: removing container" podUID="c0653ff5-b9b7-4ed9-8311-df75f12109ca" containerName="istio-proxy" Nov 03 14:47:08.304675 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.304543 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0653ff5-b9b7-4ed9-8311-df75f12109ca" containerName="istio-proxy" Nov 03 14:47:08.304675 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.304552 5060 cpu_manager.go:414] "containerMap: removing container" podUID="d073cee6-107f-4a7f-8117-f48de055e723" containerName="istio-proxy" Nov 03 14:47:08.304675 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.304556 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d073cee6-107f-4a7f-8117-f48de055e723" containerName="istio-proxy" Nov 03 14:47:08.304675 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.304567 5060 cpu_manager.go:414] "containerMap: removing container" podUID="9737ed5d-be9c-45ef-afd6-977eddcc3ce6" containerName="istio-proxy" Nov 03 14:47:08.304675 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.304572 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9737ed5d-be9c-45ef-afd6-977eddcc3ce6" containerName="istio-proxy" Nov 03 14:47:08.304675 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.304581 5060 cpu_manager.go:414] "containerMap: removing container" podUID="941df1f9-9c29-4205-b2fb-88167bd809ec" containerName="istio-proxy" Nov 03 14:47:08.304675 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.304585 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="941df1f9-9c29-4205-b2fb-88167bd809ec" containerName="istio-proxy" Nov 03 14:47:08.304675 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.304595 5060 cpu_manager.go:414] "containerMap: removing container" podUID="50a3ab6a-5b07-4d38-b00b-9b77b506ec20" containerName="istio-proxy" Nov 03 14:47:08.304675 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.304600 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50a3ab6a-5b07-4d38-b00b-9b77b506ec20" containerName="istio-proxy" Nov 03 14:47:08.304675 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.304611 5060 cpu_manager.go:414] "containerMap: removing container" podUID="43465f0f-764a-44c5-b762-9b6932953b41" containerName="istio-proxy" Nov 03 14:47:08.304675 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.304615 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43465f0f-764a-44c5-b762-9b6932953b41" containerName="istio-proxy" Nov 03 14:47:08.304675 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.304626 5060 cpu_manager.go:414] "containerMap: removing container" podUID="2d7d5e9c-625a-42af-98d7-52b9a8b5745e" containerName="istio-proxy" Nov 03 14:47:08.304675 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.304639 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d7d5e9c-625a-42af-98d7-52b9a8b5745e" containerName="istio-proxy" Nov 03 14:47:08.305504 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.304715 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d7d5e9c-625a-42af-98d7-52b9a8b5745e" containerName="istio-proxy" Nov 03 14:47:08.305504 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.304727 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="570a0e0d-16ce-486b-a762-d1e2dee1711e" containerName="cleanup" Nov 03 14:47:08.305504 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.304744 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="570a0e0d-16ce-486b-a762-d1e2dee1711e" containerName="cleanup" Nov 03 14:47:08.305504 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.304756 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0653ff5-b9b7-4ed9-8311-df75f12109ca" containerName="istio-proxy" Nov 03 14:47:08.305504 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.304767 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d073cee6-107f-4a7f-8117-f48de055e723" containerName="istio-proxy" Nov 03 14:47:08.305504 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.304778 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="9737ed5d-be9c-45ef-afd6-977eddcc3ce6" containerName="istio-proxy" Nov 03 14:47:08.305504 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.304788 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="941df1f9-9c29-4205-b2fb-88167bd809ec" containerName="istio-proxy" Nov 03 14:47:08.305504 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.304798 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="50a3ab6a-5b07-4d38-b00b-9b77b506ec20" containerName="istio-proxy" Nov 03 14:47:08.305504 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.304815 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="43465f0f-764a-44c5-b762-9b6932953b41" containerName="istio-proxy" Nov 03 14:47:08.308387 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.308361 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" Nov 03 14:47:08.318892 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.318869 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv"] Nov 03 14:47:08.319238 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.319223 5060 cpu_manager.go:414] "containerMap: removing container" podUID="d073cee6-107f-4a7f-8117-f48de055e723" containerName="istio-proxy" Nov 03 14:47:08.319323 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.319240 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d073cee6-107f-4a7f-8117-f48de055e723" containerName="istio-proxy" Nov 03 14:47:08.319381 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.319370 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="570a0e0d-16ce-486b-a762-d1e2dee1711e" containerName="cleanup" Nov 03 14:47:08.319435 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.319389 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d073cee6-107f-4a7f-8117-f48de055e723" containerName="istio-proxy" Nov 03 14:47:08.322346 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.322329 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" Nov 03 14:47:08.329985 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.329967 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4nspm\"" Nov 03 14:47:08.330100 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.329994 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 14:47:08.333638 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.333618 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm"] Nov 03 14:47:08.336896 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.336878 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" Nov 03 14:47:08.341269 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.341245 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9"] Nov 03 14:47:08.356083 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.354563 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv"] Nov 03 14:47:08.366248 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.366208 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm"] Nov 03 14:47:08.400172 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.400132 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8r2q9\" (UID: \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" Nov 03 14:47:08.400374 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.400180 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7x8hv\" (UID: \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" Nov 03 14:47:08.400374 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.400250 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8r2q9\" (UID: \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" Nov 03 14:47:08.400374 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.400327 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7x8hv\" (UID: \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" Nov 03 14:47:08.400374 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.400364 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8r2q9\" (UID: \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" Nov 03 14:47:08.400619 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.400413 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7x8hv\" (UID: \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" Nov 03 14:47:08.400619 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.400472 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptt76\" (UniqueName: \"kubernetes.io/projected/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-kube-api-access-ptt76\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8r2q9\" (UID: \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" Nov 03 14:47:08.400619 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.400512 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8r2q9\" (UID: \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" Nov 03 14:47:08.400619 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.400540 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8r2q9\" (UID: \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" Nov 03 14:47:08.400826 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.400643 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7x8hv\" (UID: \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" Nov 03 14:47:08.400826 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.400671 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8r2q9\" (UID: \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" Nov 03 14:47:08.400826 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.400768 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7x8hv\" (UID: \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" Nov 03 14:47:08.400826 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.400807 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8ldx\" (UniqueName: \"kubernetes.io/projected/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-kube-api-access-g8ldx\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7x8hv\" (UID: \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" Nov 03 14:47:08.401044 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.400893 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8r2q9\" (UID: \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" Nov 03 14:47:08.401044 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.400924 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8r2q9\" (UID: \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" Nov 03 14:47:08.401044 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.400979 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7x8hv\" (UID: \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" Nov 03 14:47:08.401228 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.401132 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7x8hv\" (UID: \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" Nov 03 14:47:08.401228 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.401189 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7x8hv\" (UID: \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" Nov 03 14:47:08.502021 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.501974 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/796dbd66-d335-4eed-bb70-073c3971adae-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qpkm\" (UID: \"796dbd66-d335-4eed-bb70-073c3971adae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" Nov 03 14:47:08.502196 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.502025 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/796dbd66-d335-4eed-bb70-073c3971adae-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qpkm\" (UID: \"796dbd66-d335-4eed-bb70-073c3971adae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" Nov 03 14:47:08.502196 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.502096 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/796dbd66-d335-4eed-bb70-073c3971adae-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qpkm\" (UID: \"796dbd66-d335-4eed-bb70-073c3971adae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" Nov 03 14:47:08.502196 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.502131 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8r2q9\" (UID: \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" Nov 03 14:47:08.502196 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.502148 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8r2q9\" (UID: \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" Nov 03 14:47:08.502342 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.502207 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/796dbd66-d335-4eed-bb70-073c3971adae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qpkm\" (UID: \"796dbd66-d335-4eed-bb70-073c3971adae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" Nov 03 14:47:08.502342 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.502257 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7x8hv\" (UID: \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" Nov 03 14:47:08.502342 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.502283 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/796dbd66-d335-4eed-bb70-073c3971adae-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qpkm\" (UID: \"796dbd66-d335-4eed-bb70-073c3971adae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" Nov 03 14:47:08.502342 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.502306 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8r2q9\" (UID: \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" Nov 03 14:47:08.502557 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.502352 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7x8hv\" (UID: \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" Nov 03 14:47:08.502557 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.502542 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8ldx\" (UniqueName: \"kubernetes.io/projected/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-kube-api-access-g8ldx\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7x8hv\" (UID: \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" Nov 03 14:47:08.502667 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.502559 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8r2q9\" (UID: \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" Nov 03 14:47:08.502667 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.502583 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8r2q9\" (UID: \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" Nov 03 14:47:08.502667 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.502612 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8r2q9\" (UID: \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" Nov 03 14:47:08.502667 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.502635 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7x8hv\" (UID: \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" Nov 03 14:47:08.502928 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.502670 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/796dbd66-d335-4eed-bb70-073c3971adae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qpkm\" (UID: \"796dbd66-d335-4eed-bb70-073c3971adae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" Nov 03 14:47:08.502928 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.502675 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8r2q9\" (UID: \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" Nov 03 14:47:08.502928 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.502699 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7x8hv\" (UID: \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" Nov 03 14:47:08.502928 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.502755 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7x8hv\" (UID: \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" Nov 03 14:47:08.502928 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.502847 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/796dbd66-d335-4eed-bb70-073c3971adae-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qpkm\" (UID: \"796dbd66-d335-4eed-bb70-073c3971adae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" Nov 03 14:47:08.502928 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.502894 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8r2q9\" (UID: \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" Nov 03 14:47:08.503244 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.502971 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7x8hv\" (UID: \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" Nov 03 14:47:08.503244 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.502998 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7x8hv\" (UID: \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" Nov 03 14:47:08.503244 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.503001 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8r2q9\" (UID: \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" Nov 03 14:47:08.503244 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.503051 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7x8hv\" (UID: \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" Nov 03 14:47:08.503244 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.503086 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8r2q9\" (UID: \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" Nov 03 14:47:08.503244 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.503109 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7x8hv\" (UID: \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" Nov 03 14:47:08.503244 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.503112 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/796dbd66-d335-4eed-bb70-073c3971adae-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qpkm\" (UID: \"796dbd66-d335-4eed-bb70-073c3971adae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" Nov 03 14:47:08.503244 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.503136 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7x8hv\" (UID: \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" Nov 03 14:47:08.503244 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.503162 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7x8hv\" (UID: \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" Nov 03 14:47:08.503244 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.503191 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwjb6\" (UniqueName: \"kubernetes.io/projected/796dbd66-d335-4eed-bb70-073c3971adae-kube-api-access-xwjb6\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qpkm\" (UID: \"796dbd66-d335-4eed-bb70-073c3971adae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" Nov 03 14:47:08.503244 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.503234 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptt76\" (UniqueName: \"kubernetes.io/projected/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-kube-api-access-ptt76\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8r2q9\" (UID: \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" Nov 03 14:47:08.503825 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.503294 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8r2q9\" (UID: \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" Nov 03 14:47:08.503825 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.503294 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7x8hv\" (UID: \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" Nov 03 14:47:08.503825 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.503305 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8r2q9\" (UID: \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" Nov 03 14:47:08.503825 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.503353 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8r2q9\" (UID: \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" Nov 03 14:47:08.503825 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.503624 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7x8hv\" (UID: \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" Nov 03 14:47:08.504754 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.504727 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8r2q9\" (UID: \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" Nov 03 14:47:08.505349 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.505328 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7x8hv\" (UID: \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" Nov 03 14:47:08.505644 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.505626 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7x8hv\" (UID: \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" Nov 03 14:47:08.505736 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.505719 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8r2q9\" (UID: \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" Nov 03 14:47:08.526990 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.526962 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7x8hv\" (UID: \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" Nov 03 14:47:08.527108 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.527008 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8ldx\" (UniqueName: \"kubernetes.io/projected/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-kube-api-access-g8ldx\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7x8hv\" (UID: \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" Nov 03 14:47:08.527153 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.527118 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptt76\" (UniqueName: \"kubernetes.io/projected/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-kube-api-access-ptt76\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8r2q9\" (UID: \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" Nov 03 14:47:08.528299 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.528274 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8r2q9\" (UID: \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" Nov 03 14:47:08.604220 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.604116 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/796dbd66-d335-4eed-bb70-073c3971adae-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qpkm\" (UID: \"796dbd66-d335-4eed-bb70-073c3971adae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" Nov 03 14:47:08.604220 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.604169 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/796dbd66-d335-4eed-bb70-073c3971adae-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qpkm\" (UID: \"796dbd66-d335-4eed-bb70-073c3971adae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" Nov 03 14:47:08.604220 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.604186 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwjb6\" (UniqueName: \"kubernetes.io/projected/796dbd66-d335-4eed-bb70-073c3971adae-kube-api-access-xwjb6\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qpkm\" (UID: \"796dbd66-d335-4eed-bb70-073c3971adae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" Nov 03 14:47:08.604220 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.604211 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/796dbd66-d335-4eed-bb70-073c3971adae-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qpkm\" (UID: \"796dbd66-d335-4eed-bb70-073c3971adae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" Nov 03 14:47:08.604220 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.604227 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/796dbd66-d335-4eed-bb70-073c3971adae-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qpkm\" (UID: \"796dbd66-d335-4eed-bb70-073c3971adae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" Nov 03 14:47:08.604623 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.604243 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/796dbd66-d335-4eed-bb70-073c3971adae-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qpkm\" (UID: \"796dbd66-d335-4eed-bb70-073c3971adae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" Nov 03 14:47:08.604623 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.604269 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/796dbd66-d335-4eed-bb70-073c3971adae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qpkm\" (UID: \"796dbd66-d335-4eed-bb70-073c3971adae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" Nov 03 14:47:08.604623 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.604306 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/796dbd66-d335-4eed-bb70-073c3971adae-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qpkm\" (UID: \"796dbd66-d335-4eed-bb70-073c3971adae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" Nov 03 14:47:08.604754 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.604625 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/796dbd66-d335-4eed-bb70-073c3971adae-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qpkm\" (UID: \"796dbd66-d335-4eed-bb70-073c3971adae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" Nov 03 14:47:08.604754 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.604648 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/796dbd66-d335-4eed-bb70-073c3971adae-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qpkm\" (UID: \"796dbd66-d335-4eed-bb70-073c3971adae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" Nov 03 14:47:08.604754 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.604704 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/796dbd66-d335-4eed-bb70-073c3971adae-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qpkm\" (UID: \"796dbd66-d335-4eed-bb70-073c3971adae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" Nov 03 14:47:08.604913 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.604792 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/796dbd66-d335-4eed-bb70-073c3971adae-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qpkm\" (UID: \"796dbd66-d335-4eed-bb70-073c3971adae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" Nov 03 14:47:08.604913 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.604821 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/796dbd66-d335-4eed-bb70-073c3971adae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qpkm\" (UID: \"796dbd66-d335-4eed-bb70-073c3971adae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" Nov 03 14:47:08.605356 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.605334 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/796dbd66-d335-4eed-bb70-073c3971adae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qpkm\" (UID: \"796dbd66-d335-4eed-bb70-073c3971adae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" Nov 03 14:47:08.606655 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.606634 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/796dbd66-d335-4eed-bb70-073c3971adae-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qpkm\" (UID: \"796dbd66-d335-4eed-bb70-073c3971adae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" Nov 03 14:47:08.606747 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.606688 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/796dbd66-d335-4eed-bb70-073c3971adae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qpkm\" (UID: \"796dbd66-d335-4eed-bb70-073c3971adae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" Nov 03 14:47:08.621711 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.621691 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" Nov 03 14:47:08.626444 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.626427 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwjb6\" (UniqueName: \"kubernetes.io/projected/796dbd66-d335-4eed-bb70-073c3971adae-kube-api-access-xwjb6\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qpkm\" (UID: \"796dbd66-d335-4eed-bb70-073c3971adae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" Nov 03 14:47:08.626646 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.626630 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/796dbd66-d335-4eed-bb70-073c3971adae-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9qpkm\" (UID: \"796dbd66-d335-4eed-bb70-073c3971adae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" Nov 03 14:47:08.634646 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.634625 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" Nov 03 14:47:08.657464 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.657434 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" Nov 03 14:47:08.944291 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.944258 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9"] Nov 03 14:47:08.947150 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.947122 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv"] Nov 03 14:47:08.947732 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:47:08.947708 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a4e5c0d_efd0_4862_91a9_7a9ba60f3c52.slice/crio-17ec538eb3886c023ed2fd568c3efb538d41ce9705f36a1d87411235969e0f90 WatchSource:0}: Error finding container 17ec538eb3886c023ed2fd568c3efb538d41ce9705f36a1d87411235969e0f90: Status 404 returned error can't find the container with id 17ec538eb3886c023ed2fd568c3efb538d41ce9705f36a1d87411235969e0f90 Nov 03 14:47:08.950367 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.950328 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:47:08.950477 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.950409 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:47:08.950477 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.950452 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:47:08.950583 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:47:08.950562 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda88f6aee_6ba7_4821_8da3_c86c4c56dcd1.slice/crio-a5173a90112f77d9b32e0e12ce96ca587033816a1e22a3dfd8a5f06b8d01e194 WatchSource:0}: Error finding container a5173a90112f77d9b32e0e12ce96ca587033816a1e22a3dfd8a5f06b8d01e194: Status 404 returned error can't find the container with id a5173a90112f77d9b32e0e12ce96ca587033816a1e22a3dfd8a5f06b8d01e194 Nov 03 14:47:08.952453 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.952414 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:47:08.952500 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.952463 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:47:08.952500 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.952491 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:47:08.986438 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.986413 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm"] Nov 03 14:47:08.988504 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:47:08.988481 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod796dbd66_d335_4eed_bb70_073c3971adae.slice/crio-15dd71d9a731d6dd357bfa223aeaa78f959184004b55f9c36c629a807b1e0d6c WatchSource:0}: Error finding container 15dd71d9a731d6dd357bfa223aeaa78f959184004b55f9c36c629a807b1e0d6c: Status 404 returned error can't find the container with id 15dd71d9a731d6dd357bfa223aeaa78f959184004b55f9c36c629a807b1e0d6c Nov 03 14:47:08.990970 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.990823 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:47:08.990970 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.990936 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:47:08.991112 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:08.990982 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:47:09.450395 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:47:09.450336 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:47:09.571338 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:09.571298 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" event={"ID":"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1","Type":"ContainerStarted","Data":"9f0a2cd5dfc0a1084149057b9ed160f146a706d081263f16e2591a48abf62b59"} Nov 03 14:47:09.572884 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:09.572376 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" event={"ID":"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1","Type":"ContainerStarted","Data":"a5173a90112f77d9b32e0e12ce96ca587033816a1e22a3dfd8a5f06b8d01e194"} Nov 03 14:47:09.578612 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:09.577918 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" event={"ID":"796dbd66-d335-4eed-bb70-073c3971adae","Type":"ContainerStarted","Data":"6a8a3b3c576c0f6a2d832c56f87cbc94d5946050f19b61eecd717a9a24629c00"} Nov 03 14:47:09.578612 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:09.577979 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" event={"ID":"796dbd66-d335-4eed-bb70-073c3971adae","Type":"ContainerStarted","Data":"15dd71d9a731d6dd357bfa223aeaa78f959184004b55f9c36c629a807b1e0d6c"} Nov 03 14:47:09.586856 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:09.580893 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" event={"ID":"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52","Type":"ContainerStarted","Data":"7f37cccf66f94ab7c4ab94e9c201318739263d25f74b25516a55295883937a57"} Nov 03 14:47:09.586856 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:09.580935 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" event={"ID":"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52","Type":"ContainerStarted","Data":"17ec538eb3886c023ed2fd568c3efb538d41ce9705f36a1d87411235969e0f90"} Nov 03 14:47:09.627860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:09.622995 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" Nov 03 14:47:09.652882 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:09.649932 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" Nov 03 14:47:09.670861 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:09.658913 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" Nov 03 14:47:09.708917 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:09.705758 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" podStartSLOduration=1.705738296 podStartE2EDuration="1.705738296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:47:09.653730581 +0000 UTC m=+2163.542520183" watchObservedRunningTime="2026-11-03 14:47:09.705738296 +0000 UTC m=+2163.594527882" Nov 03 14:47:09.729858 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:09.712967 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" podStartSLOduration=1.712946965 podStartE2EDuration="1.712946965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:47:09.70342825 +0000 UTC m=+2163.592217836" watchObservedRunningTime="2026-11-03 14:47:09.712946965 +0000 UTC m=+2163.601736561" Nov 03 14:47:09.774861 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:09.773927 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" podStartSLOduration=1.773906899 podStartE2EDuration="1.773906899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:47:09.771365958 +0000 UTC m=+2163.660155540" watchObservedRunningTime="2026-11-03 14:47:09.773906899 +0000 UTC m=+2163.662696482" Nov 03 14:47:10.623513 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:10.623473 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 03 14:47:10.624133 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:10.624099 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" podUID="3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 03 14:47:10.637902 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:10.637870 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 03 14:47:10.638141 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:10.638113 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" podUID="a88f6aee-6ba7-4821-8da3-c86c4c56dcd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 03 14:47:10.672875 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:10.672285 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 03 14:47:10.672875 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:10.672363 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" podUID="796dbd66-d335-4eed-bb70-073c3971adae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 03 14:47:11.625860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:11.625306 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 03 14:47:11.625860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:11.625375 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" podUID="3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 03 14:47:11.641862 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:11.639258 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 03 14:47:11.641862 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:11.639322 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" podUID="a88f6aee-6ba7-4821-8da3-c86c4c56dcd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 03 14:47:11.659033 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:11.658982 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 03 14:47:11.659342 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:11.659306 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" podUID="796dbd66-d335-4eed-bb70-073c3971adae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 03 14:47:12.624493 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:12.624432 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 03 14:47:12.624861 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:12.624781 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" podUID="3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 03 14:47:12.640867 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:12.637255 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 03 14:47:12.640867 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:12.637315 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" podUID="a88f6aee-6ba7-4821-8da3-c86c4c56dcd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 03 14:47:12.668859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:12.666277 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 03 14:47:12.668859 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:12.666342 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" podUID="796dbd66-d335-4eed-bb70-073c3971adae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 03 14:47:13.635858 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:13.634695 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 03 14:47:13.635858 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:13.634757 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" podUID="3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 03 14:47:13.635858 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:13.635486 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 03 14:47:13.635858 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:13.635526 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" podUID="a88f6aee-6ba7-4821-8da3-c86c4c56dcd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 03 14:47:13.659858 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:13.658163 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 03 14:47:13.659858 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:13.658223 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" podUID="796dbd66-d335-4eed-bb70-073c3971adae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 03 14:47:14.644858 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:14.644303 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 03 14:47:14.644858 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:14.644397 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" podUID="3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 03 14:47:14.648855 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:14.646197 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 03 14:47:14.648855 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:14.646250 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" podUID="a88f6aee-6ba7-4821-8da3-c86c4c56dcd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 03 14:47:14.675860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:14.664262 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 03 14:47:14.675860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:14.664358 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" podUID="796dbd66-d335-4eed-bb70-073c3971adae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 03 14:47:15.622997 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:15.622950 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 03 14:47:15.623184 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:15.623021 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" podUID="3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 03 14:47:15.635371 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:15.635339 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 03 14:47:15.635442 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:15.635412 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" podUID="a88f6aee-6ba7-4821-8da3-c86c4c56dcd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 03 14:47:15.658561 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:15.658527 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 03 14:47:15.658684 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:15.658584 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" podUID="796dbd66-d335-4eed-bb70-073c3971adae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 03 14:47:16.622844 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:16.622795 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 03 14:47:16.623260 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:16.622884 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" podUID="3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 03 14:47:16.635183 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:16.635152 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 03 14:47:16.635286 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:16.635209 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" podUID="a88f6aee-6ba7-4821-8da3-c86c4c56dcd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 03 14:47:16.658029 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:16.657998 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 03 14:47:16.658134 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:16.658085 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" podUID="796dbd66-d335-4eed-bb70-073c3971adae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 03 14:47:18.663023 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:18.662982 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:47:18.663564 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:18.663046 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" podUID="3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:47:18.677020 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:18.676979 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:47:18.677204 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:18.677057 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" podUID="a88f6aee-6ba7-4821-8da3-c86c4c56dcd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:47:18.700028 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:18.699973 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:47:18.700211 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:18.700031 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" podUID="796dbd66-d335-4eed-bb70-073c3971adae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:47:18.777393 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:18.777352 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" Nov 03 14:47:18.778310 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:18.778179 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" Nov 03 14:47:18.779195 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:18.779173 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" Nov 03 14:47:18.875142 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:18.875110 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" Nov 03 14:47:18.875756 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:18.875729 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" Nov 03 14:47:18.876627 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:18.876600 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" Nov 03 14:47:18.964813 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:18.964735 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" Nov 03 14:47:18.965972 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:18.965936 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" Nov 03 14:47:18.966565 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:47:18.966542 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" Nov 03 14:47:39.438052 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:47:39.438018 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:48:04.823862 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:48:04.822947 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:48:04.823862 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:48:04.823046 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" podUID="3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:48:04.925860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:48:04.924966 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:48:04.925860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:48:04.925062 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" podUID="a88f6aee-6ba7-4821-8da3-c86c4c56dcd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:48:05.016035 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:48:05.015977 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:48:05.016396 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:48:05.016359 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" podUID="796dbd66-d335-4eed-bb70-073c3971adae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:48:09.439169 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:48:09.439124 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:48:39.439203 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:48:39.439162 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:49:04.820080 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:49:04.820038 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:49:04.820855 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:49:04.820789 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" podUID="3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:49:04.922976 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:49:04.922935 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:49:04.923249 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:49:04.923218 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" podUID="a88f6aee-6ba7-4821-8da3-c86c4c56dcd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:49:05.014989 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:49:05.014947 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:49:05.015308 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:49:05.015276 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" podUID="796dbd66-d335-4eed-bb70-073c3971adae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:49:09.439300 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:49:09.439257 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:49:39.438192 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:49:39.438155 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:49:50.118354 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:49:50.118319 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aad267e4-gs6pz"] Nov 03 14:49:50.121937 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:49:50.121919 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad267e4-gs6pz" Nov 03 14:49:50.139105 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:49:50.139082 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdxrd\" (UniqueName: \"kubernetes.io/projected/2cdbd4ba-ddd3-422b-a4e4-7d39681495fa-kube-api-access-pdxrd\") pod \"maas-api-revoke-sub-aad267e4-gs6pz\" (UID: \"2cdbd4ba-ddd3-422b-a4e4-7d39681495fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad267e4-gs6pz" Nov 03 14:49:50.139220 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:49:50.139115 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cdbd4ba-ddd3-422b-a4e4-7d39681495fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aad267e4-gs6pz\" (UID: \"2cdbd4ba-ddd3-422b-a4e4-7d39681495fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad267e4-gs6pz" Nov 03 14:49:50.139281 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:49:50.139259 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cdbd4ba-ddd3-422b-a4e4-7d39681495fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aad267e4-gs6pz\" (UID: \"2cdbd4ba-ddd3-422b-a4e4-7d39681495fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad267e4-gs6pz" Nov 03 14:49:50.151622 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:49:50.151587 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:49:50.151959 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:49:50.151936 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\"" Nov 03 14:49:50.152138 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:49:50.151968 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:49:50.152224 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:49:50.151978 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:49:50.154306 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:49:50.154286 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aad267e4-gs6pz"] Nov 03 14:49:50.240353 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:49:50.240323 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cdbd4ba-ddd3-422b-a4e4-7d39681495fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aad267e4-gs6pz\" (UID: \"2cdbd4ba-ddd3-422b-a4e4-7d39681495fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad267e4-gs6pz" Nov 03 14:49:50.240527 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:49:50.240380 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cdbd4ba-ddd3-422b-a4e4-7d39681495fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aad267e4-gs6pz\" (UID: \"2cdbd4ba-ddd3-422b-a4e4-7d39681495fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad267e4-gs6pz" Nov 03 14:49:50.240527 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:49:50.240430 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdxrd\" (UniqueName: \"kubernetes.io/projected/2cdbd4ba-ddd3-422b-a4e4-7d39681495fa-kube-api-access-pdxrd\") pod \"maas-api-revoke-sub-aad267e4-gs6pz\" (UID: \"2cdbd4ba-ddd3-422b-a4e4-7d39681495fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad267e4-gs6pz" Nov 03 14:49:50.241023 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:49:50.241001 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cdbd4ba-ddd3-422b-a4e4-7d39681495fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aad267e4-gs6pz\" (UID: \"2cdbd4ba-ddd3-422b-a4e4-7d39681495fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad267e4-gs6pz" Nov 03 14:49:50.242823 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:49:50.242798 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cdbd4ba-ddd3-422b-a4e4-7d39681495fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aad267e4-gs6pz\" (UID: \"2cdbd4ba-ddd3-422b-a4e4-7d39681495fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad267e4-gs6pz" Nov 03 14:49:50.264173 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:49:50.264146 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdxrd\" (UniqueName: \"kubernetes.io/projected/2cdbd4ba-ddd3-422b-a4e4-7d39681495fa-kube-api-access-pdxrd\") pod \"maas-api-revoke-sub-aad267e4-gs6pz\" (UID: \"2cdbd4ba-ddd3-422b-a4e4-7d39681495fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad267e4-gs6pz" Nov 03 14:49:50.438395 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:49:50.438314 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad267e4-gs6pz" Nov 03 14:49:50.742869 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:49:50.742825 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aad267e4-gs6pz"] Nov 03 14:49:50.744990 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:49:50.744964 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cdbd4ba_ddd3_422b_a4e4_7d39681495fa.slice/crio-37035c5a228b1d35849e1d96a133510da0f02cb0d8dc8a29b6b4d8c49ae0f6f7 WatchSource:0}: Error finding container 37035c5a228b1d35849e1d96a133510da0f02cb0d8dc8a29b6b4d8c49ae0f6f7: Status 404 returned error can't find the container with id 37035c5a228b1d35849e1d96a133510da0f02cb0d8dc8a29b6b4d8c49ae0f6f7 Nov 03 14:49:51.751229 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:49:51.751144 5060 generic.go:356] "Generic (PLEG): container finished" podID="2cdbd4ba-ddd3-422b-a4e4-7d39681495fa" containerID="940ed1045c47b2b2fd507b6360f52f4a4ed1cf800142f87f01386eef04bd6f39" exitCode=0 Nov 03 14:49:51.751229 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:49:51.751202 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad267e4-gs6pz" event={"ID":"2cdbd4ba-ddd3-422b-a4e4-7d39681495fa","Type":"ContainerDied","Data":"940ed1045c47b2b2fd507b6360f52f4a4ed1cf800142f87f01386eef04bd6f39"} Nov 03 14:49:51.751630 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:49:51.751244 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad267e4-gs6pz" event={"ID":"2cdbd4ba-ddd3-422b-a4e4-7d39681495fa","Type":"ContainerStarted","Data":"37035c5a228b1d35849e1d96a133510da0f02cb0d8dc8a29b6b4d8c49ae0f6f7"} Nov 03 14:49:52.889141 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:49:52.889114 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad267e4-gs6pz" Nov 03 14:49:52.967655 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:49:52.967627 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cdbd4ba-ddd3-422b-a4e4-7d39681495fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cdbd4ba-ddd3-422b-a4e4-7d39681495fa-maas-api-xks-ca\") pod \"2cdbd4ba-ddd3-422b-a4e4-7d39681495fa\" (UID: \"2cdbd4ba-ddd3-422b-a4e4-7d39681495fa\") " Nov 03 14:49:52.967807 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:49:52.967669 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cdbd4ba-ddd3-422b-a4e4-7d39681495fa-kube-api-access-pdxrd\" (UniqueName: \"kubernetes.io/projected/2cdbd4ba-ddd3-422b-a4e4-7d39681495fa-kube-api-access-pdxrd\") pod \"2cdbd4ba-ddd3-422b-a4e4-7d39681495fa\" (UID: \"2cdbd4ba-ddd3-422b-a4e4-7d39681495fa\") " Nov 03 14:49:52.967807 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:49:52.967723 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cdbd4ba-ddd3-422b-a4e4-7d39681495fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cdbd4ba-ddd3-422b-a4e4-7d39681495fa-maas-api-openshift-ca\") pod \"2cdbd4ba-ddd3-422b-a4e4-7d39681495fa\" (UID: \"2cdbd4ba-ddd3-422b-a4e4-7d39681495fa\") " Nov 03 14:49:52.968173 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:49:52.968141 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cdbd4ba-ddd3-422b-a4e4-7d39681495fa-maas-api-openshift-ca" pod "2cdbd4ba-ddd3-422b-a4e4-7d39681495fa" (UID: "2cdbd4ba-ddd3-422b-a4e4-7d39681495fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:49:52.969690 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:49:52.969662 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cdbd4ba-ddd3-422b-a4e4-7d39681495fa-maas-api-xks-ca" pod "2cdbd4ba-ddd3-422b-a4e4-7d39681495fa" (UID: "2cdbd4ba-ddd3-422b-a4e4-7d39681495fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:49:52.969807 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:49:52.969783 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cdbd4ba-ddd3-422b-a4e4-7d39681495fa-kube-api-access-pdxrd" pod "2cdbd4ba-ddd3-422b-a4e4-7d39681495fa" (UID: "2cdbd4ba-ddd3-422b-a4e4-7d39681495fa"). InnerVolumeSpecName "kube-api-access-pdxrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:49:53.068705 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:49:53.068635 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cdbd4ba-ddd3-422b-a4e4-7d39681495fa-maas-api-xks-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:49:53.068705 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:49:53.068658 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdxrd\" (UniqueName: \"kubernetes.io/projected/2cdbd4ba-ddd3-422b-a4e4-7d39681495fa-kube-api-access-pdxrd\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:49:53.068705 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:49:53.068667 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cdbd4ba-ddd3-422b-a4e4-7d39681495fa-maas-api-openshift-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:49:53.763502 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:49:53.763472 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad267e4-gs6pz" Nov 03 14:49:53.763698 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:49:53.763504 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad267e4-gs6pz" event={"ID":"2cdbd4ba-ddd3-422b-a4e4-7d39681495fa","Type":"ContainerDied","Data":"37035c5a228b1d35849e1d96a133510da0f02cb0d8dc8a29b6b4d8c49ae0f6f7"} Nov 03 14:49:53.763698 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:49:53.763550 5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37035c5a228b1d35849e1d96a133510da0f02cb0d8dc8a29b6b4d8c49ae0f6f7" Nov 03 14:50:05.054693 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:05.054652 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90b2f64b-fwlcj"] Nov 03 14:50:05.055277 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:05.055255 5060 cpu_manager.go:414] "containerMap: removing container" podUID="2cdbd4ba-ddd3-422b-a4e4-7d39681495fa" containerName="revoke-keys" Nov 03 14:50:05.055321 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:05.055280 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cdbd4ba-ddd3-422b-a4e4-7d39681495fa" containerName="revoke-keys" Nov 03 14:50:05.055447 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:05.055435 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cdbd4ba-ddd3-422b-a4e4-7d39681495fa" containerName="revoke-keys" Nov 03 14:50:05.058773 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:05.058753 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90b2f64b-fwlcj" Nov 03 14:50:05.080289 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:05.080247 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90b2f64b-fwlcj"] Nov 03 14:50:05.090662 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:05.090627 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:50:05.090662 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:05.090632 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:50:05.090662 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:05.090639 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:50:05.090922 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:05.090688 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\"" Nov 03 14:50:05.176686 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:05.176644 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v9v6\" (UniqueName: \"kubernetes.io/projected/a6a2d239-c45b-4b26-8f10-7d9c4817065b-kube-api-access-6v9v6\") pod \"maas-api-revoke-sub-90b2f64b-fwlcj\" (UID: \"a6a2d239-c45b-4b26-8f10-7d9c4817065b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90b2f64b-fwlcj" Nov 03 14:50:05.176686 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:05.176686 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6a2d239-c45b-4b26-8f10-7d9c4817065b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90b2f64b-fwlcj\" (UID: \"a6a2d239-c45b-4b26-8f10-7d9c4817065b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90b2f64b-fwlcj" Nov 03 14:50:05.177021 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:05.176973 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6a2d239-c45b-4b26-8f10-7d9c4817065b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90b2f64b-fwlcj\" (UID: \"a6a2d239-c45b-4b26-8f10-7d9c4817065b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90b2f64b-fwlcj" Nov 03 14:50:05.278475 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:05.278434 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6a2d239-c45b-4b26-8f10-7d9c4817065b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90b2f64b-fwlcj\" (UID: \"a6a2d239-c45b-4b26-8f10-7d9c4817065b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90b2f64b-fwlcj" Nov 03 14:50:05.278656 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:05.278518 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v9v6\" (UniqueName: \"kubernetes.io/projected/a6a2d239-c45b-4b26-8f10-7d9c4817065b-kube-api-access-6v9v6\") pod \"maas-api-revoke-sub-90b2f64b-fwlcj\" (UID: \"a6a2d239-c45b-4b26-8f10-7d9c4817065b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90b2f64b-fwlcj" Nov 03 14:50:05.278656 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:05.278543 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6a2d239-c45b-4b26-8f10-7d9c4817065b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90b2f64b-fwlcj\" (UID: \"a6a2d239-c45b-4b26-8f10-7d9c4817065b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90b2f64b-fwlcj" Nov 03 14:50:05.279241 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:05.279219 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6a2d239-c45b-4b26-8f10-7d9c4817065b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90b2f64b-fwlcj\" (UID: \"a6a2d239-c45b-4b26-8f10-7d9c4817065b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90b2f64b-fwlcj" Nov 03 14:50:05.280852 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:05.280816 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6a2d239-c45b-4b26-8f10-7d9c4817065b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90b2f64b-fwlcj\" (UID: \"a6a2d239-c45b-4b26-8f10-7d9c4817065b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90b2f64b-fwlcj" Nov 03 14:50:05.302845 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:05.302811 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v9v6\" (UniqueName: \"kubernetes.io/projected/a6a2d239-c45b-4b26-8f10-7d9c4817065b-kube-api-access-6v9v6\") pod \"maas-api-revoke-sub-90b2f64b-fwlcj\" (UID: \"a6a2d239-c45b-4b26-8f10-7d9c4817065b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90b2f64b-fwlcj" Nov 03 14:50:05.372028 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:05.371942 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90b2f64b-fwlcj" Nov 03 14:50:05.696304 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:05.696268 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90b2f64b-fwlcj"] Nov 03 14:50:05.830761 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:05.830715 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90b2f64b-fwlcj" event={"ID":"a6a2d239-c45b-4b26-8f10-7d9c4817065b","Type":"ContainerStarted","Data":"26699e258c9dd04cb02c0635b98d8e9adfde0e160e98305ba8caebf75285e0a3"} Nov 03 14:50:06.878913 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:06.878088 5060 generic.go:356] "Generic (PLEG): container finished" podID="a6a2d239-c45b-4b26-8f10-7d9c4817065b" containerID="e19a8fbbd0ef793ac77a5f4af7f777cace9feaccae9ae09b071a35cbd973e47e" exitCode=0 Nov 03 14:50:06.878913 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:06.878208 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90b2f64b-fwlcj" event={"ID":"a6a2d239-c45b-4b26-8f10-7d9c4817065b","Type":"ContainerDied","Data":"e19a8fbbd0ef793ac77a5f4af7f777cace9feaccae9ae09b071a35cbd973e47e"} Nov 03 14:50:08.394229 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:08.394194 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90b2f64b-fwlcj" Nov 03 14:50:08.577186 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:08.576263 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6a2d239-c45b-4b26-8f10-7d9c4817065b-kube-api-access-6v9v6\" (UniqueName: \"kubernetes.io/projected/a6a2d239-c45b-4b26-8f10-7d9c4817065b-kube-api-access-6v9v6\") pod \"a6a2d239-c45b-4b26-8f10-7d9c4817065b\" (UID: \"a6a2d239-c45b-4b26-8f10-7d9c4817065b\") " Nov 03 14:50:08.577186 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:08.576385 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6a2d239-c45b-4b26-8f10-7d9c4817065b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6a2d239-c45b-4b26-8f10-7d9c4817065b-maas-api-openshift-ca\") pod \"a6a2d239-c45b-4b26-8f10-7d9c4817065b\" (UID: \"a6a2d239-c45b-4b26-8f10-7d9c4817065b\") " Nov 03 14:50:08.577186 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:08.576427 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6a2d239-c45b-4b26-8f10-7d9c4817065b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6a2d239-c45b-4b26-8f10-7d9c4817065b-maas-api-xks-ca\") pod \"a6a2d239-c45b-4b26-8f10-7d9c4817065b\" (UID: \"a6a2d239-c45b-4b26-8f10-7d9c4817065b\") " Nov 03 14:50:08.590268 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:08.590197 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6a2d239-c45b-4b26-8f10-7d9c4817065b-maas-api-xks-ca" pod "a6a2d239-c45b-4b26-8f10-7d9c4817065b" (UID: "a6a2d239-c45b-4b26-8f10-7d9c4817065b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:50:08.590865 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:08.590811 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6a2d239-c45b-4b26-8f10-7d9c4817065b-maas-api-openshift-ca" pod "a6a2d239-c45b-4b26-8f10-7d9c4817065b" (UID: "a6a2d239-c45b-4b26-8f10-7d9c4817065b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:50:08.612399 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:08.609344 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6a2d239-c45b-4b26-8f10-7d9c4817065b-kube-api-access-6v9v6" pod "a6a2d239-c45b-4b26-8f10-7d9c4817065b" (UID: "a6a2d239-c45b-4b26-8f10-7d9c4817065b"). InnerVolumeSpecName "kube-api-access-6v9v6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:50:08.681869 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:08.679409 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v9v6\" (UniqueName: \"kubernetes.io/projected/a6a2d239-c45b-4b26-8f10-7d9c4817065b-kube-api-access-6v9v6\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:50:08.681869 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:08.679457 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6a2d239-c45b-4b26-8f10-7d9c4817065b-maas-api-openshift-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:50:08.681869 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:08.679474 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6a2d239-c45b-4b26-8f10-7d9c4817065b-maas-api-xks-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:50:08.892014 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:08.891913 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90b2f64b-fwlcj" Nov 03 14:50:08.892014 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:08.891931 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90b2f64b-fwlcj" event={"ID":"a6a2d239-c45b-4b26-8f10-7d9c4817065b","Type":"ContainerDied","Data":"26699e258c9dd04cb02c0635b98d8e9adfde0e160e98305ba8caebf75285e0a3"} Nov 03 14:50:08.892014 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:08.892019 5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26699e258c9dd04cb02c0635b98d8e9adfde0e160e98305ba8caebf75285e0a3" Nov 03 14:50:09.439764 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:50:09.439728 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:50:39.441536 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:50:39.441486 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:50:40.702051 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:40.702013 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-a58e3e/simulator-single-main-a58e3e-kserve-59d6845567-fzjwh"] Nov 03 14:50:40.702454 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:40.702413 5060 cpu_manager.go:414] "containerMap: removing container" podUID="a6a2d239-c45b-4b26-8f10-7d9c4817065b" containerName="revoke-keys" Nov 03 14:50:40.702454 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:40.702440 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6a2d239-c45b-4b26-8f10-7d9c4817065b" containerName="revoke-keys" Nov 03 14:50:40.702540 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:40.702529 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6a2d239-c45b-4b26-8f10-7d9c4817065b" containerName="revoke-keys" Nov 03 14:50:40.705666 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:40.705648 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a58e3e/simulator-single-main-a58e3e-kserve-59d6845567-fzjwh" Nov 03 14:50:40.724510 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:40.724491 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-a58e3e\"/\"kube-root-ca.crt\"" Nov 03 14:50:40.736943 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:40.736914 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-a58e3e\"/\"simulator-single-main-a58e3e-kserve-self-signed-certs\"" Nov 03 14:50:40.737077 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:40.736945 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-a58e3e\"/\"default-dockercfg-tw98d\"" Nov 03 14:50:40.737077 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:40.736963 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-a58e3e\"/\"openshift-service-ca.crt\"" Nov 03 14:50:40.740490 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:40.740470 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-a58e3e/simulator-single-main-a58e3e-kserve-59d6845567-fzjwh"] Nov 03 14:50:40.808275 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:40.808236 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b68ce00c-4172-487f-a759-dcabe7867212-model-cache\") pod \"simulator-single-main-a58e3e-kserve-59d6845567-fzjwh\" (UID: \"b68ce00c-4172-487f-a759-dcabe7867212\") " pod="e2e-models-e2e-worker-single-main-a58e3e/simulator-single-main-a58e3e-kserve-59d6845567-fzjwh" Nov 03 14:50:40.808439 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:40.808289 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b68ce00c-4172-487f-a759-dcabe7867212-home\") pod \"simulator-single-main-a58e3e-kserve-59d6845567-fzjwh\" (UID: \"b68ce00c-4172-487f-a759-dcabe7867212\") " pod="e2e-models-e2e-worker-single-main-a58e3e/simulator-single-main-a58e3e-kserve-59d6845567-fzjwh" Nov 03 14:50:40.808439 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:40.808406 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b68ce00c-4172-487f-a759-dcabe7867212-tls-certs\") pod \"simulator-single-main-a58e3e-kserve-59d6845567-fzjwh\" (UID: \"b68ce00c-4172-487f-a759-dcabe7867212\") " pod="e2e-models-e2e-worker-single-main-a58e3e/simulator-single-main-a58e3e-kserve-59d6845567-fzjwh" Nov 03 14:50:40.808520 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:40.808449 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdhs5\" (UniqueName: \"kubernetes.io/projected/b68ce00c-4172-487f-a759-dcabe7867212-kube-api-access-qdhs5\") pod \"simulator-single-main-a58e3e-kserve-59d6845567-fzjwh\" (UID: \"b68ce00c-4172-487f-a759-dcabe7867212\") " pod="e2e-models-e2e-worker-single-main-a58e3e/simulator-single-main-a58e3e-kserve-59d6845567-fzjwh" Nov 03 14:50:40.808556 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:40.808547 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b68ce00c-4172-487f-a759-dcabe7867212-tmp-dir\") pod \"simulator-single-main-a58e3e-kserve-59d6845567-fzjwh\" (UID: \"b68ce00c-4172-487f-a759-dcabe7867212\") " pod="e2e-models-e2e-worker-single-main-a58e3e/simulator-single-main-a58e3e-kserve-59d6845567-fzjwh" Nov 03 14:50:40.808590 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:40.808573 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b68ce00c-4172-487f-a759-dcabe7867212-dshm\") pod \"simulator-single-main-a58e3e-kserve-59d6845567-fzjwh\" (UID: \"b68ce00c-4172-487f-a759-dcabe7867212\") " pod="e2e-models-e2e-worker-single-main-a58e3e/simulator-single-main-a58e3e-kserve-59d6845567-fzjwh" Nov 03 14:50:40.909755 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:40.909717 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b68ce00c-4172-487f-a759-dcabe7867212-dshm\") pod \"simulator-single-main-a58e3e-kserve-59d6845567-fzjwh\" (UID: \"b68ce00c-4172-487f-a759-dcabe7867212\") " pod="e2e-models-e2e-worker-single-main-a58e3e/simulator-single-main-a58e3e-kserve-59d6845567-fzjwh" Nov 03 14:50:40.909980 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:40.909775 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b68ce00c-4172-487f-a759-dcabe7867212-model-cache\") pod \"simulator-single-main-a58e3e-kserve-59d6845567-fzjwh\" (UID: \"b68ce00c-4172-487f-a759-dcabe7867212\") " pod="e2e-models-e2e-worker-single-main-a58e3e/simulator-single-main-a58e3e-kserve-59d6845567-fzjwh" Nov 03 14:50:40.909980 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:40.909797 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b68ce00c-4172-487f-a759-dcabe7867212-home\") pod \"simulator-single-main-a58e3e-kserve-59d6845567-fzjwh\" (UID: \"b68ce00c-4172-487f-a759-dcabe7867212\") " pod="e2e-models-e2e-worker-single-main-a58e3e/simulator-single-main-a58e3e-kserve-59d6845567-fzjwh" Nov 03 14:50:40.909980 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:40.909856 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b68ce00c-4172-487f-a759-dcabe7867212-tls-certs\") pod \"simulator-single-main-a58e3e-kserve-59d6845567-fzjwh\" (UID: \"b68ce00c-4172-487f-a759-dcabe7867212\") " pod="e2e-models-e2e-worker-single-main-a58e3e/simulator-single-main-a58e3e-kserve-59d6845567-fzjwh" Nov 03 14:50:40.909980 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:40.909893 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdhs5\" (UniqueName: \"kubernetes.io/projected/b68ce00c-4172-487f-a759-dcabe7867212-kube-api-access-qdhs5\") pod \"simulator-single-main-a58e3e-kserve-59d6845567-fzjwh\" (UID: \"b68ce00c-4172-487f-a759-dcabe7867212\") " pod="e2e-models-e2e-worker-single-main-a58e3e/simulator-single-main-a58e3e-kserve-59d6845567-fzjwh" Nov 03 14:50:40.909980 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:40.909930 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b68ce00c-4172-487f-a759-dcabe7867212-tmp-dir\") pod \"simulator-single-main-a58e3e-kserve-59d6845567-fzjwh\" (UID: \"b68ce00c-4172-487f-a759-dcabe7867212\") " pod="e2e-models-e2e-worker-single-main-a58e3e/simulator-single-main-a58e3e-kserve-59d6845567-fzjwh" Nov 03 14:50:40.910283 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:40.910214 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b68ce00c-4172-487f-a759-dcabe7867212-model-cache\") pod \"simulator-single-main-a58e3e-kserve-59d6845567-fzjwh\" (UID: \"b68ce00c-4172-487f-a759-dcabe7867212\") " pod="e2e-models-e2e-worker-single-main-a58e3e/simulator-single-main-a58e3e-kserve-59d6845567-fzjwh" Nov 03 14:50:40.910283 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:40.910258 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b68ce00c-4172-487f-a759-dcabe7867212-home\") pod \"simulator-single-main-a58e3e-kserve-59d6845567-fzjwh\" (UID: \"b68ce00c-4172-487f-a759-dcabe7867212\") " pod="e2e-models-e2e-worker-single-main-a58e3e/simulator-single-main-a58e3e-kserve-59d6845567-fzjwh" Nov 03 14:50:40.910370 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:40.910315 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b68ce00c-4172-487f-a759-dcabe7867212-tmp-dir\") pod \"simulator-single-main-a58e3e-kserve-59d6845567-fzjwh\" (UID: \"b68ce00c-4172-487f-a759-dcabe7867212\") " pod="e2e-models-e2e-worker-single-main-a58e3e/simulator-single-main-a58e3e-kserve-59d6845567-fzjwh" Nov 03 14:50:40.912162 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:40.912137 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b68ce00c-4172-487f-a759-dcabe7867212-dshm\") pod \"simulator-single-main-a58e3e-kserve-59d6845567-fzjwh\" (UID: \"b68ce00c-4172-487f-a759-dcabe7867212\") " pod="e2e-models-e2e-worker-single-main-a58e3e/simulator-single-main-a58e3e-kserve-59d6845567-fzjwh" Nov 03 14:50:40.912366 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:40.912349 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b68ce00c-4172-487f-a759-dcabe7867212-tls-certs\") pod \"simulator-single-main-a58e3e-kserve-59d6845567-fzjwh\" (UID: \"b68ce00c-4172-487f-a759-dcabe7867212\") " pod="e2e-models-e2e-worker-single-main-a58e3e/simulator-single-main-a58e3e-kserve-59d6845567-fzjwh" Nov 03 14:50:40.931734 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:40.931710 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdhs5\" (UniqueName: \"kubernetes.io/projected/b68ce00c-4172-487f-a759-dcabe7867212-kube-api-access-qdhs5\") pod \"simulator-single-main-a58e3e-kserve-59d6845567-fzjwh\" (UID: \"b68ce00c-4172-487f-a759-dcabe7867212\") " pod="e2e-models-e2e-worker-single-main-a58e3e/simulator-single-main-a58e3e-kserve-59d6845567-fzjwh" Nov 03 14:50:41.033479 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:41.033444 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a58e3e/simulator-single-main-a58e3e-kserve-59d6845567-fzjwh" Nov 03 14:50:41.343936 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:41.343910 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-a58e3e/simulator-single-main-a58e3e-kserve-59d6845567-fzjwh"] Nov 03 14:50:41.346720 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:50:41.346690 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb68ce00c_4172_487f_a759_dcabe7867212.slice/crio-c0d7ebc120cf6c05400ee06f91d8e44035d4afb86b7004921466ff6cbe827102 WatchSource:0}: Error finding container c0d7ebc120cf6c05400ee06f91d8e44035d4afb86b7004921466ff6cbe827102: Status 404 returned error can't find the container with id c0d7ebc120cf6c05400ee06f91d8e44035d4afb86b7004921466ff6cbe827102 Nov 03 14:50:42.102593 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:42.102551 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a58e3e/simulator-single-main-a58e3e-kserve-59d6845567-fzjwh" event={"ID":"b68ce00c-4172-487f-a759-dcabe7867212","Type":"ContainerStarted","Data":"6776c913eeaa7435ce775cbe3062aec6ccd41b7b4045aa76ad9c1bcdf32bc57e"} Nov 03 14:50:42.102999 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:42.102603 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a58e3e/simulator-single-main-a58e3e-kserve-59d6845567-fzjwh" event={"ID":"b68ce00c-4172-487f-a759-dcabe7867212","Type":"ContainerStarted","Data":"c0d7ebc120cf6c05400ee06f91d8e44035d4afb86b7004921466ff6cbe827102"} Nov 03 14:50:42.148492 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:42.148443 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-a58e3e/simulator-single-main-a58e3e-kserve-59d6845567-fzjwh" podStartSLOduration=2.148429714 podStartE2EDuration="2.148429714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:50:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:50:42.146397428 +0000 UTC m=+2376.035187011" watchObservedRunningTime="2026-11-03 14:50:42.148429714 +0000 UTC m=+2376.037219294" Nov 03 14:50:43.034171 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:43.034127 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-a58e3e/simulator-single-main-a58e3e-kserve-59d6845567-fzjwh" Nov 03 14:50:51.034367 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:51.034279 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-a58e3e/simulator-single-main-a58e3e-kserve-59d6845567-fzjwh" Nov 03 14:50:51.046722 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:51.046695 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-a58e3e/simulator-single-main-a58e3e-kserve-59d6845567-fzjwh" Nov 03 14:50:51.160963 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:50:51.160934 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-a58e3e/simulator-single-main-a58e3e-kserve-59d6845567-fzjwh" Nov 03 14:51:07.495700 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:51:07.495658 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/3.log" Nov 03 14:51:07.496204 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:51:07.495788 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/3.log" Nov 03 14:51:09.439778 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:51:09.439737 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:51:28.343446 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:51:28.343405 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-a58e3e/simulator-single-main-a58e3e-kserve-59d6845567-fzjwh"] Nov 03 14:51:39.438518 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:51:39.438473 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:51:43.364270 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:51:43.364231 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-a58e3e/simulator-single-main-a58e3e-kserve-59d6845567-fzjwh" podUID="b68ce00c-4172-487f-a759-dcabe7867212" containerName="main" containerID="cri-o://6776c913eeaa7435ce775cbe3062aec6ccd41b7b4045aa76ad9c1bcdf32bc57e" gracePeriod=45 Nov 03 14:51:43.630213 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:51:43.630142 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a58e3e/simulator-single-main-a58e3e-kserve-59d6845567-fzjwh" Nov 03 14:51:43.794213 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:51:43.794180 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b68ce00c-4172-487f-a759-dcabe7867212-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b68ce00c-4172-487f-a759-dcabe7867212-tmp-dir\") pod \"b68ce00c-4172-487f-a759-dcabe7867212\" (UID: \"b68ce00c-4172-487f-a759-dcabe7867212\") " Nov 03 14:51:43.794382 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:51:43.794220 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b68ce00c-4172-487f-a759-dcabe7867212-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b68ce00c-4172-487f-a759-dcabe7867212-dshm\") pod \"b68ce00c-4172-487f-a759-dcabe7867212\" (UID: \"b68ce00c-4172-487f-a759-dcabe7867212\") " Nov 03 14:51:43.794382 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:51:43.794307 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b68ce00c-4172-487f-a759-dcabe7867212-kube-api-access-qdhs5\" (UniqueName: \"kubernetes.io/projected/b68ce00c-4172-487f-a759-dcabe7867212-kube-api-access-qdhs5\") pod \"b68ce00c-4172-487f-a759-dcabe7867212\" (UID: \"b68ce00c-4172-487f-a759-dcabe7867212\") " Nov 03 14:51:43.794382 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:51:43.794339 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b68ce00c-4172-487f-a759-dcabe7867212-tls-certs\" (UniqueName: \"kubernetes.io/secret/b68ce00c-4172-487f-a759-dcabe7867212-tls-certs\") pod \"b68ce00c-4172-487f-a759-dcabe7867212\" (UID: \"b68ce00c-4172-487f-a759-dcabe7867212\") " Nov 03 14:51:43.794550 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:51:43.794402 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b68ce00c-4172-487f-a759-dcabe7867212-home\" (UniqueName: \"kubernetes.io/empty-dir/b68ce00c-4172-487f-a759-dcabe7867212-home\") pod \"b68ce00c-4172-487f-a759-dcabe7867212\" (UID: \"b68ce00c-4172-487f-a759-dcabe7867212\") " Nov 03 14:51:43.794550 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:51:43.794446 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b68ce00c-4172-487f-a759-dcabe7867212-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b68ce00c-4172-487f-a759-dcabe7867212-model-cache\") pod \"b68ce00c-4172-487f-a759-dcabe7867212\" (UID: \"b68ce00c-4172-487f-a759-dcabe7867212\") " Nov 03 14:51:43.794550 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:51:43.794487 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b68ce00c-4172-487f-a759-dcabe7867212-tmp-dir" pod "b68ce00c-4172-487f-a759-dcabe7867212" (UID: "b68ce00c-4172-487f-a759-dcabe7867212"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:51:43.794750 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:51:43.794720 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b68ce00c-4172-487f-a759-dcabe7867212-home" pod "b68ce00c-4172-487f-a759-dcabe7867212" (UID: "b68ce00c-4172-487f-a759-dcabe7867212"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:51:43.794809 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:51:43.794741 5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b68ce00c-4172-487f-a759-dcabe7867212-tmp-dir\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:51:43.794809 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:51:43.794730 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b68ce00c-4172-487f-a759-dcabe7867212-model-cache" pod "b68ce00c-4172-487f-a759-dcabe7867212" (UID: "b68ce00c-4172-487f-a759-dcabe7867212"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:51:43.796526 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:51:43.796500 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b68ce00c-4172-487f-a759-dcabe7867212-dshm" pod "b68ce00c-4172-487f-a759-dcabe7867212" (UID: "b68ce00c-4172-487f-a759-dcabe7867212"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:51:43.796638 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:51:43.796577 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b68ce00c-4172-487f-a759-dcabe7867212-tls-certs" pod "b68ce00c-4172-487f-a759-dcabe7867212" (UID: "b68ce00c-4172-487f-a759-dcabe7867212"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:51:43.796680 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:51:43.796660 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b68ce00c-4172-487f-a759-dcabe7867212-kube-api-access-qdhs5" pod "b68ce00c-4172-487f-a759-dcabe7867212" (UID: "b68ce00c-4172-487f-a759-dcabe7867212"). InnerVolumeSpecName "kube-api-access-qdhs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:51:43.895289 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:51:43.895218 5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b68ce00c-4172-487f-a759-dcabe7867212-dshm\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:51:43.895289 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:51:43.895245 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdhs5\" (UniqueName: \"kubernetes.io/projected/b68ce00c-4172-487f-a759-dcabe7867212-kube-api-access-qdhs5\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:51:43.895289 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:51:43.895256 5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b68ce00c-4172-487f-a759-dcabe7867212-tls-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:51:43.895289 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:51:43.895265 5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b68ce00c-4172-487f-a759-dcabe7867212-home\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:51:43.895289 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:51:43.895272 5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b68ce00c-4172-487f-a759-dcabe7867212-model-cache\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:51:44.435043 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:51:44.435013 5060 generic.go:356] "Generic (PLEG): container finished" podID="b68ce00c-4172-487f-a759-dcabe7867212" containerID="6776c913eeaa7435ce775cbe3062aec6ccd41b7b4045aa76ad9c1bcdf32bc57e" exitCode=0 Nov 03 14:51:44.435461 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:51:44.435054 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a58e3e/simulator-single-main-a58e3e-kserve-59d6845567-fzjwh" Nov 03 14:51:44.435461 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:51:44.435096 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a58e3e/simulator-single-main-a58e3e-kserve-59d6845567-fzjwh" event={"ID":"b68ce00c-4172-487f-a759-dcabe7867212","Type":"ContainerDied","Data":"6776c913eeaa7435ce775cbe3062aec6ccd41b7b4045aa76ad9c1bcdf32bc57e"} Nov 03 14:51:44.435461 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:51:44.435144 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a58e3e/simulator-single-main-a58e3e-kserve-59d6845567-fzjwh" event={"ID":"b68ce00c-4172-487f-a759-dcabe7867212","Type":"ContainerDied","Data":"c0d7ebc120cf6c05400ee06f91d8e44035d4afb86b7004921466ff6cbe827102"} Nov 03 14:51:44.435461 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:51:44.435165 5060 scope.go:122] "RemoveContainer" containerID="6776c913eeaa7435ce775cbe3062aec6ccd41b7b4045aa76ad9c1bcdf32bc57e" Nov 03 14:51:44.444023 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:51:44.444004 5060 scope.go:122] "RemoveContainer" containerID="6776c913eeaa7435ce775cbe3062aec6ccd41b7b4045aa76ad9c1bcdf32bc57e" Nov 03 14:51:44.444278 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:51:44.444259 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6776c913eeaa7435ce775cbe3062aec6ccd41b7b4045aa76ad9c1bcdf32bc57e\": container with ID starting with 6776c913eeaa7435ce775cbe3062aec6ccd41b7b4045aa76ad9c1bcdf32bc57e not found: ID does not exist" containerID="6776c913eeaa7435ce775cbe3062aec6ccd41b7b4045aa76ad9c1bcdf32bc57e" Nov 03 14:51:44.444332 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:51:44.444292 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6776c913eeaa7435ce775cbe3062aec6ccd41b7b4045aa76ad9c1bcdf32bc57e"} err="failed to get container status \"6776c913eeaa7435ce775cbe3062aec6ccd41b7b4045aa76ad9c1bcdf32bc57e\": rpc error: code = NotFound desc = could not find container \"6776c913eeaa7435ce775cbe3062aec6ccd41b7b4045aa76ad9c1bcdf32bc57e\": container with ID starting with 6776c913eeaa7435ce775cbe3062aec6ccd41b7b4045aa76ad9c1bcdf32bc57e not found: ID does not exist" Nov 03 14:51:44.493184 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:51:44.493153 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-a58e3e/simulator-single-main-a58e3e-kserve-59d6845567-fzjwh"] Nov 03 14:51:44.499990 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:51:44.499961 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-a58e3e/simulator-single-main-a58e3e-kserve-59d6845567-fzjwh"] Nov 03 14:51:46.254412 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:51:46.254371 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b68ce00c-4172-487f-a759-dcabe7867212" path="/var/lib/kubelet/pods/b68ce00c-4172-487f-a759-dcabe7867212/volumes" Nov 03 14:52:09.440430 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:52:09.440387 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:52:39.522859 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:52:39.522079 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:53:09.441177 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:53:09.441141 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:53:39.438105 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:53:39.438065 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:53:49.826916 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:53:49.825922 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:53:49.826916 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:53:49.825994 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" podUID="3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:53:49.925860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:53:49.925754 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:53:49.925860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:53:49.925828 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" podUID="a88f6aee-6ba7-4821-8da3-c86c4c56dcd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:53:50.018860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:53:50.017966 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:53:50.018860 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:53:50.018054 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" podUID="796dbd66-d335-4eed-bb70-073c3971adae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:54:09.438690 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:54:09.438653 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:54:39.441561 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:54:39.441511 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:54:53.053058 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:54:53.053016 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aad267e4-gs6pz"] Nov 03 14:54:53.065740 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:54:53.065706 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aad267e4-gs6pz"] Nov 03 14:54:54.247877 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:54:54.247829 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cdbd4ba-ddd3-422b-a4e4-7d39681495fa" path="/var/lib/kubelet/pods/2cdbd4ba-ddd3-422b-a4e4-7d39681495fa/volumes" Nov 03 14:55:09.040769 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:55:09.040728 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90b2f64b-fwlcj"] Nov 03 14:55:09.053354 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:55:09.053325 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90b2f64b-fwlcj"] Nov 03 14:55:09.439034 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:55:09.438939 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:55:10.247972 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:55:10.247940 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6a2d239-c45b-4b26-8f10-7d9c4817065b" path="/var/lib/kubelet/pods/a6a2d239-c45b-4b26-8f10-7d9c4817065b/volumes" Nov 03 14:55:16.805235 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:55:16.805177 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-72b663/ipp-route-72b663-kserve-7fb7d986df-bzx2j"] Nov 03 14:55:16.806093 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:55:16.805810 5060 cpu_manager.go:414] "containerMap: removing container" podUID="b68ce00c-4172-487f-a759-dcabe7867212" containerName="main" Nov 03 14:55:16.806093 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:55:16.805827 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b68ce00c-4172-487f-a759-dcabe7867212" containerName="main" Nov 03 14:55:16.806093 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:55:16.805930 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="b68ce00c-4172-487f-a759-dcabe7867212" containerName="main" Nov 03 14:55:16.809419 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:55:16.809401 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-72b663/ipp-route-72b663-kserve-7fb7d986df-bzx2j" Nov 03 14:55:16.826689 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:55:16.826667 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-72b663\"/\"kube-root-ca.crt\"" Nov 03 14:55:16.838578 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:55:16.838554 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-72b663\"/\"ipp-route-72b663-kserve-self-signed-certs\"" Nov 03 14:55:16.839728 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:55:16.839710 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-72b663\"/\"default-dockercfg-p8t87\"" Nov 03 14:55:16.840508 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:55:16.840487 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-72b663/ipp-route-72b663-kserve-7fb7d986df-bzx2j"] Nov 03 14:55:16.926799 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:55:16.926764 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-72b663\"/\"openshift-service-ca.crt\"" Nov 03 14:55:16.960588 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:55:16.960551 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f842518-c98d-44f6-acc5-0819e4e9ba67-model-cache\") pod \"ipp-route-72b663-kserve-7fb7d986df-bzx2j\" (UID: \"0f842518-c98d-44f6-acc5-0819e4e9ba67\") " pod="ai-tenant-e2e-ipp-a-72b663/ipp-route-72b663-kserve-7fb7d986df-bzx2j" Nov 03 14:55:16.960767 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:55:16.960680 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f842518-c98d-44f6-acc5-0819e4e9ba67-tmp-dir\") pod \"ipp-route-72b663-kserve-7fb7d986df-bzx2j\" (UID: \"0f842518-c98d-44f6-acc5-0819e4e9ba67\") " pod="ai-tenant-e2e-ipp-a-72b663/ipp-route-72b663-kserve-7fb7d986df-bzx2j" Nov 03 14:55:16.960767 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:55:16.960709 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f842518-c98d-44f6-acc5-0819e4e9ba67-home\") pod \"ipp-route-72b663-kserve-7fb7d986df-bzx2j\" (UID: \"0f842518-c98d-44f6-acc5-0819e4e9ba67\") " pod="ai-tenant-e2e-ipp-a-72b663/ipp-route-72b663-kserve-7fb7d986df-bzx2j" Nov 03 14:55:16.960893 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:55:16.960780 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f842518-c98d-44f6-acc5-0819e4e9ba67-tls-certs\") pod \"ipp-route-72b663-kserve-7fb7d986df-bzx2j\" (UID: \"0f842518-c98d-44f6-acc5-0819e4e9ba67\") " pod="ai-tenant-e2e-ipp-a-72b663/ipp-route-72b663-kserve-7fb7d986df-bzx2j" Nov 03 14:55:16.960893 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:55:16.960814 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f842518-c98d-44f6-acc5-0819e4e9ba67-dshm\") pod \"ipp-route-72b663-kserve-7fb7d986df-bzx2j\" (UID: \"0f842518-c98d-44f6-acc5-0819e4e9ba67\") " pod="ai-tenant-e2e-ipp-a-72b663/ipp-route-72b663-kserve-7fb7d986df-bzx2j" Nov 03 14:55:16.960972 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:55:16.960893 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bnqp\" (UniqueName: \"kubernetes.io/projected/0f842518-c98d-44f6-acc5-0819e4e9ba67-kube-api-access-9bnqp\") pod \"ipp-route-72b663-kserve-7fb7d986df-bzx2j\" (UID: \"0f842518-c98d-44f6-acc5-0819e4e9ba67\") " pod="ai-tenant-e2e-ipp-a-72b663/ipp-route-72b663-kserve-7fb7d986df-bzx2j" Nov 03 14:55:17.062285 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:55:17.062204 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f842518-c98d-44f6-acc5-0819e4e9ba67-tmp-dir\") pod \"ipp-route-72b663-kserve-7fb7d986df-bzx2j\" (UID: \"0f842518-c98d-44f6-acc5-0819e4e9ba67\") " pod="ai-tenant-e2e-ipp-a-72b663/ipp-route-72b663-kserve-7fb7d986df-bzx2j" Nov 03 14:55:17.062285 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:55:17.062241 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f842518-c98d-44f6-acc5-0819e4e9ba67-home\") pod \"ipp-route-72b663-kserve-7fb7d986df-bzx2j\" (UID: \"0f842518-c98d-44f6-acc5-0819e4e9ba67\") " pod="ai-tenant-e2e-ipp-a-72b663/ipp-route-72b663-kserve-7fb7d986df-bzx2j" Nov 03 14:55:17.062285 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:55:17.062259 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f842518-c98d-44f6-acc5-0819e4e9ba67-tls-certs\") pod \"ipp-route-72b663-kserve-7fb7d986df-bzx2j\" (UID: \"0f842518-c98d-44f6-acc5-0819e4e9ba67\") " pod="ai-tenant-e2e-ipp-a-72b663/ipp-route-72b663-kserve-7fb7d986df-bzx2j" Nov 03 14:55:17.062285 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:55:17.062278 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f842518-c98d-44f6-acc5-0819e4e9ba67-dshm\") pod \"ipp-route-72b663-kserve-7fb7d986df-bzx2j\" (UID: \"0f842518-c98d-44f6-acc5-0819e4e9ba67\") " pod="ai-tenant-e2e-ipp-a-72b663/ipp-route-72b663-kserve-7fb7d986df-bzx2j" Nov 03 14:55:17.062611 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:55:17.062305 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bnqp\" (UniqueName: \"kubernetes.io/projected/0f842518-c98d-44f6-acc5-0819e4e9ba67-kube-api-access-9bnqp\") pod \"ipp-route-72b663-kserve-7fb7d986df-bzx2j\" (UID: \"0f842518-c98d-44f6-acc5-0819e4e9ba67\") " pod="ai-tenant-e2e-ipp-a-72b663/ipp-route-72b663-kserve-7fb7d986df-bzx2j" Nov 03 14:55:17.062611 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:55:17.062354 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f842518-c98d-44f6-acc5-0819e4e9ba67-model-cache\") pod \"ipp-route-72b663-kserve-7fb7d986df-bzx2j\" (UID: \"0f842518-c98d-44f6-acc5-0819e4e9ba67\") " pod="ai-tenant-e2e-ipp-a-72b663/ipp-route-72b663-kserve-7fb7d986df-bzx2j" Nov 03 14:55:17.062726 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:55:17.062680 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f842518-c98d-44f6-acc5-0819e4e9ba67-tmp-dir\") pod \"ipp-route-72b663-kserve-7fb7d986df-bzx2j\" (UID: \"0f842518-c98d-44f6-acc5-0819e4e9ba67\") " pod="ai-tenant-e2e-ipp-a-72b663/ipp-route-72b663-kserve-7fb7d986df-bzx2j" Nov 03 14:55:17.062726 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:55:17.062696 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f842518-c98d-44f6-acc5-0819e4e9ba67-home\") pod \"ipp-route-72b663-kserve-7fb7d986df-bzx2j\" (UID: \"0f842518-c98d-44f6-acc5-0819e4e9ba67\") " pod="ai-tenant-e2e-ipp-a-72b663/ipp-route-72b663-kserve-7fb7d986df-bzx2j" Nov 03 14:55:17.062869 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:55:17.062763 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f842518-c98d-44f6-acc5-0819e4e9ba67-model-cache\") pod \"ipp-route-72b663-kserve-7fb7d986df-bzx2j\" (UID: \"0f842518-c98d-44f6-acc5-0819e4e9ba67\") " pod="ai-tenant-e2e-ipp-a-72b663/ipp-route-72b663-kserve-7fb7d986df-bzx2j" Nov 03 14:55:17.064593 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:55:17.064577 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f842518-c98d-44f6-acc5-0819e4e9ba67-dshm\") pod \"ipp-route-72b663-kserve-7fb7d986df-bzx2j\" (UID: \"0f842518-c98d-44f6-acc5-0819e4e9ba67\") " pod="ai-tenant-e2e-ipp-a-72b663/ipp-route-72b663-kserve-7fb7d986df-bzx2j" Nov 03 14:55:17.064716 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:55:17.064699 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f842518-c98d-44f6-acc5-0819e4e9ba67-tls-certs\") pod \"ipp-route-72b663-kserve-7fb7d986df-bzx2j\" (UID: \"0f842518-c98d-44f6-acc5-0819e4e9ba67\") " pod="ai-tenant-e2e-ipp-a-72b663/ipp-route-72b663-kserve-7fb7d986df-bzx2j" Nov 03 14:55:17.084771 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:55:17.084747 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bnqp\" (UniqueName: \"kubernetes.io/projected/0f842518-c98d-44f6-acc5-0819e4e9ba67-kube-api-access-9bnqp\") pod \"ipp-route-72b663-kserve-7fb7d986df-bzx2j\" (UID: \"0f842518-c98d-44f6-acc5-0819e4e9ba67\") " pod="ai-tenant-e2e-ipp-a-72b663/ipp-route-72b663-kserve-7fb7d986df-bzx2j" Nov 03 14:55:17.134842 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:55:17.134773 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-72b663/ipp-route-72b663-kserve-7fb7d986df-bzx2j" Nov 03 14:55:17.448471 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:55:17.448427 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-72b663/ipp-route-72b663-kserve-7fb7d986df-bzx2j"] Nov 03 14:55:17.450348 ip-10-0-143-171 kubenswrapper[5060]: W1103 14:55:17.450321 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f842518_c98d_44f6_acc5_0819e4e9ba67.slice/crio-6bc53a8512b292b402e1b9015083e42f83e3b3dd7f7c44eeebbad8eab71851f8 WatchSource:0}: Error finding container 6bc53a8512b292b402e1b9015083e42f83e3b3dd7f7c44eeebbad8eab71851f8: Status 404 returned error can't find the container with id 6bc53a8512b292b402e1b9015083e42f83e3b3dd7f7c44eeebbad8eab71851f8 Nov 03 14:55:17.763091 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:55:17.763055 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-72b663/ipp-route-72b663-kserve-7fb7d986df-bzx2j" event={"ID":"0f842518-c98d-44f6-acc5-0819e4e9ba67","Type":"ContainerStarted","Data":"012a977d9d731b352bbb4b487a4fe82af8768a1c1784df7949603ceeec59920d"} Nov 03 14:55:17.763248 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:55:17.763103 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-72b663/ipp-route-72b663-kserve-7fb7d986df-bzx2j" event={"ID":"0f842518-c98d-44f6-acc5-0819e4e9ba67","Type":"ContainerStarted","Data":"6bc53a8512b292b402e1b9015083e42f83e3b3dd7f7c44eeebbad8eab71851f8"} Nov 03 14:55:17.807562 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:55:17.807503 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-72b663/ipp-route-72b663-kserve-7fb7d986df-bzx2j" podStartSLOduration=1.807486254 podStartE2EDuration="1.807486254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:55:17.80662585 +0000 UTC m=+2651.695415430" watchObservedRunningTime="2026-11-03 14:55:17.807486254 +0000 UTC m=+2651.696275836" Nov 03 14:55:18.135251 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:55:18.135165 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-72b663/ipp-route-72b663-kserve-7fb7d986df-bzx2j" Nov 03 14:55:27.134932 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:55:27.134886 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-72b663/ipp-route-72b663-kserve-7fb7d986df-bzx2j" Nov 03 14:55:27.147076 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:55:27.147055 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-72b663/ipp-route-72b663-kserve-7fb7d986df-bzx2j" Nov 03 14:55:27.826657 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:55:27.826629 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-72b663/ipp-route-72b663-kserve-7fb7d986df-bzx2j" Nov 03 14:55:39.441893 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:55:39.441771 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:56:00.501038 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:56:00.500997 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-72b663/ipp-route-72b663-kserve-7fb7d986df-bzx2j"] Nov 03 14:56:07.538338 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:56:07.538300 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/3.log" Nov 03 14:56:07.540903 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:56:07.540882 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/3.log" Nov 03 14:56:08.447359 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:56:08.447319 5060 scope.go:122] "RemoveContainer" containerID="e19a8fbbd0ef793ac77a5f4af7f777cace9feaccae9ae09b071a35cbd973e47e" Nov 03 14:56:08.456257 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:56:08.456239 5060 scope.go:122] "RemoveContainer" containerID="940ed1045c47b2b2fd507b6360f52f4a4ed1cf800142f87f01386eef04bd6f39" Nov 03 14:56:09.438903 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:56:09.438867 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:56:15.521458 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:56:15.521393 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-72b663/ipp-route-72b663-kserve-7fb7d986df-bzx2j" podUID="0f842518-c98d-44f6-acc5-0819e4e9ba67" containerName="main" containerID="cri-o://012a977d9d731b352bbb4b487a4fe82af8768a1c1784df7949603ceeec59920d" gracePeriod=45 Nov 03 14:56:15.776118 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:56:15.776063 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-72b663/ipp-route-72b663-kserve-7fb7d986df-bzx2j" Nov 03 14:56:15.801632 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:56:15.801606 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f842518-c98d-44f6-acc5-0819e4e9ba67-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f842518-c98d-44f6-acc5-0819e4e9ba67-tmp-dir\") pod \"0f842518-c98d-44f6-acc5-0819e4e9ba67\" (UID: \"0f842518-c98d-44f6-acc5-0819e4e9ba67\") " Nov 03 14:56:15.801741 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:56:15.801640 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f842518-c98d-44f6-acc5-0819e4e9ba67-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f842518-c98d-44f6-acc5-0819e4e9ba67-dshm\") pod \"0f842518-c98d-44f6-acc5-0819e4e9ba67\" (UID: \"0f842518-c98d-44f6-acc5-0819e4e9ba67\") " Nov 03 14:56:15.801741 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:56:15.801672 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f842518-c98d-44f6-acc5-0819e4e9ba67-home\" (UniqueName: \"kubernetes.io/empty-dir/0f842518-c98d-44f6-acc5-0819e4e9ba67-home\") pod \"0f842518-c98d-44f6-acc5-0819e4e9ba67\" (UID: \"0f842518-c98d-44f6-acc5-0819e4e9ba67\") " Nov 03 14:56:15.801741 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:56:15.801730 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f842518-c98d-44f6-acc5-0819e4e9ba67-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f842518-c98d-44f6-acc5-0819e4e9ba67-model-cache\") pod \"0f842518-c98d-44f6-acc5-0819e4e9ba67\" (UID: \"0f842518-c98d-44f6-acc5-0819e4e9ba67\") " Nov 03 14:56:15.801869 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:56:15.801771 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f842518-c98d-44f6-acc5-0819e4e9ba67-tls-certs\" (UniqueName: \"kubernetes.io/secret/0f842518-c98d-44f6-acc5-0819e4e9ba67-tls-certs\") pod \"0f842518-c98d-44f6-acc5-0819e4e9ba67\" (UID: \"0f842518-c98d-44f6-acc5-0819e4e9ba67\") " Nov 03 14:56:15.801909 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:56:15.801868 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f842518-c98d-44f6-acc5-0819e4e9ba67-kube-api-access-9bnqp\" (UniqueName: \"kubernetes.io/projected/0f842518-c98d-44f6-acc5-0819e4e9ba67-kube-api-access-9bnqp\") pod \"0f842518-c98d-44f6-acc5-0819e4e9ba67\" (UID: \"0f842518-c98d-44f6-acc5-0819e4e9ba67\") " Nov 03 14:56:15.802012 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:56:15.801865 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f842518-c98d-44f6-acc5-0819e4e9ba67-tmp-dir" pod "0f842518-c98d-44f6-acc5-0819e4e9ba67" (UID: "0f842518-c98d-44f6-acc5-0819e4e9ba67"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:56:15.802012 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:56:15.801997 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f842518-c98d-44f6-acc5-0819e4e9ba67-home" pod "0f842518-c98d-44f6-acc5-0819e4e9ba67" (UID: "0f842518-c98d-44f6-acc5-0819e4e9ba67"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:56:15.802127 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:56:15.802012 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f842518-c98d-44f6-acc5-0819e4e9ba67-model-cache" pod "0f842518-c98d-44f6-acc5-0819e4e9ba67" (UID: "0f842518-c98d-44f6-acc5-0819e4e9ba67"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:56:15.802217 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:56:15.802199 5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f842518-c98d-44f6-acc5-0819e4e9ba67-tmp-dir\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:56:15.802260 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:56:15.802223 5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f842518-c98d-44f6-acc5-0819e4e9ba67-home\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:56:15.802260 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:56:15.802237 5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f842518-c98d-44f6-acc5-0819e4e9ba67-model-cache\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:56:15.803774 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:56:15.803753 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f842518-c98d-44f6-acc5-0819e4e9ba67-dshm" pod "0f842518-c98d-44f6-acc5-0819e4e9ba67" (UID: "0f842518-c98d-44f6-acc5-0819e4e9ba67"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:56:15.803880 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:56:15.803863 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f842518-c98d-44f6-acc5-0819e4e9ba67-kube-api-access-9bnqp" pod "0f842518-c98d-44f6-acc5-0819e4e9ba67" (UID: "0f842518-c98d-44f6-acc5-0819e4e9ba67"). InnerVolumeSpecName "kube-api-access-9bnqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:56:15.803951 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:56:15.803925 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f842518-c98d-44f6-acc5-0819e4e9ba67-tls-certs" pod "0f842518-c98d-44f6-acc5-0819e4e9ba67" (UID: "0f842518-c98d-44f6-acc5-0819e4e9ba67"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:56:15.902773 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:56:15.902740 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bnqp\" (UniqueName: \"kubernetes.io/projected/0f842518-c98d-44f6-acc5-0819e4e9ba67-kube-api-access-9bnqp\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:56:15.902773 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:56:15.902768 5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f842518-c98d-44f6-acc5-0819e4e9ba67-dshm\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:56:15.902773 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:56:15.902778 5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f842518-c98d-44f6-acc5-0819e4e9ba67-tls-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 14:56:16.068659 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:56:16.068572 5060 generic.go:356] "Generic (PLEG): container finished" podID="0f842518-c98d-44f6-acc5-0819e4e9ba67" containerID="012a977d9d731b352bbb4b487a4fe82af8768a1c1784df7949603ceeec59920d" exitCode=0 Nov 03 14:56:16.068659 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:56:16.068623 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-72b663/ipp-route-72b663-kserve-7fb7d986df-bzx2j" Nov 03 14:56:16.068917 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:56:16.068660 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-72b663/ipp-route-72b663-kserve-7fb7d986df-bzx2j" event={"ID":"0f842518-c98d-44f6-acc5-0819e4e9ba67","Type":"ContainerDied","Data":"012a977d9d731b352bbb4b487a4fe82af8768a1c1784df7949603ceeec59920d"} Nov 03 14:56:16.068917 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:56:16.068709 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-72b663/ipp-route-72b663-kserve-7fb7d986df-bzx2j" event={"ID":"0f842518-c98d-44f6-acc5-0819e4e9ba67","Type":"ContainerDied","Data":"6bc53a8512b292b402e1b9015083e42f83e3b3dd7f7c44eeebbad8eab71851f8"} Nov 03 14:56:16.068917 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:56:16.068727 5060 scope.go:122] "RemoveContainer" containerID="012a977d9d731b352bbb4b487a4fe82af8768a1c1784df7949603ceeec59920d" Nov 03 14:56:16.078308 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:56:16.078288 5060 scope.go:122] "RemoveContainer" containerID="012a977d9d731b352bbb4b487a4fe82af8768a1c1784df7949603ceeec59920d" Nov 03 14:56:16.078608 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:56:16.078586 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"012a977d9d731b352bbb4b487a4fe82af8768a1c1784df7949603ceeec59920d\": container with ID starting with 012a977d9d731b352bbb4b487a4fe82af8768a1c1784df7949603ceeec59920d not found: ID does not exist" containerID="012a977d9d731b352bbb4b487a4fe82af8768a1c1784df7949603ceeec59920d" Nov 03 14:56:16.078696 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:56:16.078616 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"012a977d9d731b352bbb4b487a4fe82af8768a1c1784df7949603ceeec59920d"} err="failed to get container status \"012a977d9d731b352bbb4b487a4fe82af8768a1c1784df7949603ceeec59920d\": rpc error: code = NotFound desc = could not find container \"012a977d9d731b352bbb4b487a4fe82af8768a1c1784df7949603ceeec59920d\": container with ID starting with 012a977d9d731b352bbb4b487a4fe82af8768a1c1784df7949603ceeec59920d not found: ID does not exist" Nov 03 14:56:16.125886 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:56:16.125860 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-72b663/ipp-route-72b663-kserve-7fb7d986df-bzx2j"] Nov 03 14:56:16.130618 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:56:16.130596 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-72b663/ipp-route-72b663-kserve-7fb7d986df-bzx2j"] Nov 03 14:56:16.248085 ip-10-0-143-171 kubenswrapper[5060]: I1103 14:56:16.248055 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f842518-c98d-44f6-acc5-0819e4e9ba67" path="/var/lib/kubelet/pods/0f842518-c98d-44f6-acc5-0819e4e9ba67/volumes" Nov 03 14:56:39.438804 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:56:39.438770 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:57:09.439370 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:57:09.439332 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:57:39.438372 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:57:39.438330 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:58:09.453084 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:58:09.453024 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:58:39.437876 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:58:39.437812 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:59:09.437798 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:59:09.437762 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 14:59:39.438947 ip-10-0-143-171 kubenswrapper[5060]: E1103 14:59:39.438908 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:00:09.438997 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:00:09.438962 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:00:17.037239 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:00:17.037202 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mnlct"] Nov 03 15:00:17.037735 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:00:17.037662 5060 cpu_manager.go:414] "containerMap: removing container" podUID="0f842518-c98d-44f6-acc5-0819e4e9ba67" containerName="main" Nov 03 15:00:17.037735 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:00:17.037679 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f842518-c98d-44f6-acc5-0819e4e9ba67" containerName="main" Nov 03 15:00:17.037808 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:00:17.037772 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f842518-c98d-44f6-acc5-0819e4e9ba67" containerName="main" Nov 03 15:00:17.040984 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:00:17.040968 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mnlct" Nov 03 15:00:17.072869 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:00:17.072822 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:00:17.072869 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:00:17.072852 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 15:00:17.073088 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:00:17.072942 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\"" Nov 03 15:00:17.073088 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:00:17.072978 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 15:00:17.076599 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:00:17.076570 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mnlct"] Nov 03 15:00:17.110714 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:00:17.110687 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5383fccc-ffe8-48f9-9145-b4729cbe466f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-mnlct\" (UID: \"5383fccc-ffe8-48f9-9145-b4729cbe466f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mnlct" Nov 03 15:00:17.110714 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:00:17.110714 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlnjx\" (UniqueName: \"kubernetes.io/projected/5383fccc-ffe8-48f9-9145-b4729cbe466f-kube-api-access-dlnjx\") pod \"maas-api-revoke-sub-2e1f5b5f-mnlct\" (UID: \"5383fccc-ffe8-48f9-9145-b4729cbe466f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mnlct" Nov 03 15:00:17.110898 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:00:17.110807 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5383fccc-ffe8-48f9-9145-b4729cbe466f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-mnlct\" (UID: \"5383fccc-ffe8-48f9-9145-b4729cbe466f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mnlct" Nov 03 15:00:17.212204 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:00:17.212167 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5383fccc-ffe8-48f9-9145-b4729cbe466f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-mnlct\" (UID: \"5383fccc-ffe8-48f9-9145-b4729cbe466f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mnlct" Nov 03 15:00:17.212204 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:00:17.212203 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlnjx\" (UniqueName: \"kubernetes.io/projected/5383fccc-ffe8-48f9-9145-b4729cbe466f-kube-api-access-dlnjx\") pod \"maas-api-revoke-sub-2e1f5b5f-mnlct\" (UID: \"5383fccc-ffe8-48f9-9145-b4729cbe466f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mnlct" Nov 03 15:00:17.212423 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:00:17.212247 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5383fccc-ffe8-48f9-9145-b4729cbe466f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-mnlct\" (UID: \"5383fccc-ffe8-48f9-9145-b4729cbe466f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mnlct" Nov 03 15:00:17.212867 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:00:17.212812 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5383fccc-ffe8-48f9-9145-b4729cbe466f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-mnlct\" (UID: \"5383fccc-ffe8-48f9-9145-b4729cbe466f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mnlct" Nov 03 15:00:17.214568 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:00:17.214548 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5383fccc-ffe8-48f9-9145-b4729cbe466f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-mnlct\" (UID: \"5383fccc-ffe8-48f9-9145-b4729cbe466f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mnlct" Nov 03 15:00:17.234789 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:00:17.234765 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlnjx\" (UniqueName: \"kubernetes.io/projected/5383fccc-ffe8-48f9-9145-b4729cbe466f-kube-api-access-dlnjx\") pod \"maas-api-revoke-sub-2e1f5b5f-mnlct\" (UID: \"5383fccc-ffe8-48f9-9145-b4729cbe466f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mnlct" Nov 03 15:00:17.353410 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:00:17.353339 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mnlct" Nov 03 15:00:17.656576 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:00:17.656552 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mnlct"] Nov 03 15:00:17.658749 ip-10-0-143-171 kubenswrapper[5060]: W1103 15:00:17.658722 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5383fccc_ffe8_48f9_9145_b4729cbe466f.slice/crio-7c4e07642060cdf07012d4ee9cebf8fc27be66da315a8341ba2c7fed3f67a394 WatchSource:0}: Error finding container 7c4e07642060cdf07012d4ee9cebf8fc27be66da315a8341ba2c7fed3f67a394: Status 404 returned error can't find the container with id 7c4e07642060cdf07012d4ee9cebf8fc27be66da315a8341ba2c7fed3f67a394 Nov 03 15:00:18.452433 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:00:18.452394 5060 generic.go:356] "Generic (PLEG): container finished" podID="5383fccc-ffe8-48f9-9145-b4729cbe466f" containerID="3fa39f29cae522c50b720469a60876f18244d22d2132a82242a8a9833b08a013" exitCode=0 Nov 03 15:00:18.452787 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:00:18.452470 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mnlct" event={"ID":"5383fccc-ffe8-48f9-9145-b4729cbe466f","Type":"ContainerDied","Data":"3fa39f29cae522c50b720469a60876f18244d22d2132a82242a8a9833b08a013"} Nov 03 15:00:18.452787 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:00:18.452551 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mnlct" event={"ID":"5383fccc-ffe8-48f9-9145-b4729cbe466f","Type":"ContainerStarted","Data":"7c4e07642060cdf07012d4ee9cebf8fc27be66da315a8341ba2c7fed3f67a394"} Nov 03 15:00:19.589410 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:00:19.589381 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mnlct" Nov 03 15:00:19.634444 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:00:19.634415 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5383fccc-ffe8-48f9-9145-b4729cbe466f-kube-api-access-dlnjx\" (UniqueName: \"kubernetes.io/projected/5383fccc-ffe8-48f9-9145-b4729cbe466f-kube-api-access-dlnjx\") pod \"5383fccc-ffe8-48f9-9145-b4729cbe466f\" (UID: \"5383fccc-ffe8-48f9-9145-b4729cbe466f\") " Nov 03 15:00:19.634615 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:00:19.634497 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5383fccc-ffe8-48f9-9145-b4729cbe466f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5383fccc-ffe8-48f9-9145-b4729cbe466f-maas-api-openshift-ca\") pod \"5383fccc-ffe8-48f9-9145-b4729cbe466f\" (UID: \"5383fccc-ffe8-48f9-9145-b4729cbe466f\") " Nov 03 15:00:19.634615 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:00:19.634541 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5383fccc-ffe8-48f9-9145-b4729cbe466f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5383fccc-ffe8-48f9-9145-b4729cbe466f-maas-api-xks-ca\") pod \"5383fccc-ffe8-48f9-9145-b4729cbe466f\" (UID: \"5383fccc-ffe8-48f9-9145-b4729cbe466f\") " Nov 03 15:00:19.634946 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:00:19.634916 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5383fccc-ffe8-48f9-9145-b4729cbe466f-maas-api-openshift-ca" pod "5383fccc-ffe8-48f9-9145-b4729cbe466f" (UID: "5383fccc-ffe8-48f9-9145-b4729cbe466f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:00:19.636920 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:00:19.636893 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5383fccc-ffe8-48f9-9145-b4729cbe466f-kube-api-access-dlnjx" pod "5383fccc-ffe8-48f9-9145-b4729cbe466f" (UID: "5383fccc-ffe8-48f9-9145-b4729cbe466f"). InnerVolumeSpecName "kube-api-access-dlnjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:00:19.637625 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:00:19.637589 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5383fccc-ffe8-48f9-9145-b4729cbe466f-maas-api-xks-ca" pod "5383fccc-ffe8-48f9-9145-b4729cbe466f" (UID: "5383fccc-ffe8-48f9-9145-b4729cbe466f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:00:19.735341 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:00:19.735274 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlnjx\" (UniqueName: \"kubernetes.io/projected/5383fccc-ffe8-48f9-9145-b4729cbe466f-kube-api-access-dlnjx\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 15:00:19.735341 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:00:19.735299 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5383fccc-ffe8-48f9-9145-b4729cbe466f-maas-api-openshift-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 15:00:19.735341 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:00:19.735309 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5383fccc-ffe8-48f9-9145-b4729cbe466f-maas-api-xks-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 15:00:20.464765 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:00:20.464731 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mnlct" Nov 03 15:00:20.464765 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:00:20.464758 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mnlct" event={"ID":"5383fccc-ffe8-48f9-9145-b4729cbe466f","Type":"ContainerDied","Data":"7c4e07642060cdf07012d4ee9cebf8fc27be66da315a8341ba2c7fed3f67a394"} Nov 03 15:00:20.464993 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:00:20.464805 5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c4e07642060cdf07012d4ee9cebf8fc27be66da315a8341ba2c7fed3f67a394" Nov 03 15:00:39.438415 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:00:39.438378 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:01:07.572764 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:01:07.572733 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/3.log" Nov 03 15:01:07.576425 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:01:07.576396 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/3.log" Nov 03 15:01:09.440620 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:01:09.440577 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:01:39.438191 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:01:39.438147 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:02:09.441126 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:02:09.441091 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:02:39.438776 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:02:39.438739 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:02:49.828914 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:02:49.827928 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:02:49.828914 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:02:49.828033 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" podUID="3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:02:49.931857 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:02:49.927922 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:02:49.931857 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:02:49.928003 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" podUID="a88f6aee-6ba7-4821-8da3-c86c4c56dcd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:02:50.018859 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:02:50.017948 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:02:50.018859 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:02:50.018035 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" podUID="796dbd66-d335-4eed-bb70-073c3971adae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:03:09.438284 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:03:09.438245 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:03:39.439131 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:03:39.439075 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:04:09.439186 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:04:09.439137 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:04:39.439190 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:04:39.439148 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:05:09.440385 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:05:09.440343 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:05:20.043423 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:05:20.043391 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mnlct"] Nov 03 15:05:20.056175 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:05:20.056145 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-mnlct"] Nov 03 15:05:20.248566 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:05:20.248531 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5383fccc-ffe8-48f9-9145-b4729cbe466f" path="/var/lib/kubelet/pods/5383fccc-ffe8-48f9-9145-b4729cbe466f/volumes" Nov 03 15:05:39.438155 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:05:39.438118 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:06:07.607121 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:06:07.607088 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/3.log" Nov 03 15:06:07.612002 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:06:07.611980 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/3.log" Nov 03 15:06:09.439166 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:06:09.439130 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:06:27.498507 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:06:27.498461 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-82pkj"] Nov 03 15:06:27.499114 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:06:27.498728 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-82pkj" podUID="b0d2f12e-9c6c-4e22-ba6f-197f411b0666" containerName="manager" containerID="cri-o://ef04332a24640b1550a8bd7c96589061aa07f08c6ee0ca1942f3d5c5e90f1039" gracePeriod=10 Nov 03 15:06:27.956225 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:06:27.956200 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-82pkj" Nov 03 15:06:28.013135 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:06:28.013044 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0d2f12e-9c6c-4e22-ba6f-197f411b0666-kube-api-access-jzgpl\" (UniqueName: \"kubernetes.io/projected/b0d2f12e-9c6c-4e22-ba6f-197f411b0666-kube-api-access-jzgpl\") pod \"b0d2f12e-9c6c-4e22-ba6f-197f411b0666\" (UID: \"b0d2f12e-9c6c-4e22-ba6f-197f411b0666\") " Nov 03 15:06:28.013292 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:06:28.013197 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0d2f12e-9c6c-4e22-ba6f-197f411b0666-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0d2f12e-9c6c-4e22-ba6f-197f411b0666-extensions-socket-volume\") pod \"b0d2f12e-9c6c-4e22-ba6f-197f411b0666\" (UID: \"b0d2f12e-9c6c-4e22-ba6f-197f411b0666\") " Nov 03 15:06:28.013593 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:06:28.013565 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d2f12e-9c6c-4e22-ba6f-197f411b0666-extensions-socket-volume" pod "b0d2f12e-9c6c-4e22-ba6f-197f411b0666" (UID: "b0d2f12e-9c6c-4e22-ba6f-197f411b0666"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:06:28.015120 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:06:28.015097 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0d2f12e-9c6c-4e22-ba6f-197f411b0666-kube-api-access-jzgpl" pod "b0d2f12e-9c6c-4e22-ba6f-197f411b0666" (UID: "b0d2f12e-9c6c-4e22-ba6f-197f411b0666"). InnerVolumeSpecName "kube-api-access-jzgpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:06:28.114622 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:06:28.114574 5060 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0d2f12e-9c6c-4e22-ba6f-197f411b0666-extensions-socket-volume\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 15:06:28.114622 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:06:28.114612 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzgpl\" (UniqueName: \"kubernetes.io/projected/b0d2f12e-9c6c-4e22-ba6f-197f411b0666-kube-api-access-jzgpl\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 15:06:28.797502 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:06:28.797464 5060 generic.go:356] "Generic (PLEG): container finished" podID="b0d2f12e-9c6c-4e22-ba6f-197f411b0666" containerID="ef04332a24640b1550a8bd7c96589061aa07f08c6ee0ca1942f3d5c5e90f1039" exitCode=0 Nov 03 15:06:28.797968 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:06:28.797517 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-82pkj" Nov 03 15:06:28.797968 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:06:28.797532 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-82pkj" event={"ID":"b0d2f12e-9c6c-4e22-ba6f-197f411b0666","Type":"ContainerDied","Data":"ef04332a24640b1550a8bd7c96589061aa07f08c6ee0ca1942f3d5c5e90f1039"} Nov 03 15:06:28.797968 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:06:28.797584 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-82pkj" event={"ID":"b0d2f12e-9c6c-4e22-ba6f-197f411b0666","Type":"ContainerDied","Data":"74cfecfd02be6266f26c1009dbf65b1459b0b11cb877cde286aa3ab5a13aef97"} Nov 03 15:06:28.797968 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:06:28.797616 5060 scope.go:122] "RemoveContainer" containerID="ef04332a24640b1550a8bd7c96589061aa07f08c6ee0ca1942f3d5c5e90f1039" Nov 03 15:06:28.810323 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:06:28.810304 5060 scope.go:122] "RemoveContainer" containerID="ef04332a24640b1550a8bd7c96589061aa07f08c6ee0ca1942f3d5c5e90f1039" Nov 03 15:06:28.810594 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:06:28.810571 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef04332a24640b1550a8bd7c96589061aa07f08c6ee0ca1942f3d5c5e90f1039\": container with ID starting with ef04332a24640b1550a8bd7c96589061aa07f08c6ee0ca1942f3d5c5e90f1039 not found: ID does not exist" containerID="ef04332a24640b1550a8bd7c96589061aa07f08c6ee0ca1942f3d5c5e90f1039" Nov 03 15:06:28.810649 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:06:28.810603 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef04332a24640b1550a8bd7c96589061aa07f08c6ee0ca1942f3d5c5e90f1039"} err="failed to get container status \"ef04332a24640b1550a8bd7c96589061aa07f08c6ee0ca1942f3d5c5e90f1039\": rpc error: code = NotFound desc = could not find container \"ef04332a24640b1550a8bd7c96589061aa07f08c6ee0ca1942f3d5c5e90f1039\": container with ID starting with ef04332a24640b1550a8bd7c96589061aa07f08c6ee0ca1942f3d5c5e90f1039 not found: ID does not exist" Nov 03 15:06:28.869940 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:06:28.869896 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-82pkj"] Nov 03 15:06:28.882636 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:06:28.882595 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-82pkj"] Nov 03 15:06:30.247499 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:06:30.247463 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0d2f12e-9c6c-4e22-ba6f-197f411b0666" path="/var/lib/kubelet/pods/b0d2f12e-9c6c-4e22-ba6f-197f411b0666/volumes" Nov 03 15:06:39.438918 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:06:39.438882 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:07:08.598784 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:07:08.598755 5060 scope.go:122] "RemoveContainer" containerID="3fa39f29cae522c50b720469a60876f18244d22d2132a82242a8a9833b08a013" Nov 03 15:07:09.438741 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:07:09.438701 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:07:30.701661 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:07:30.701623 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5rnng"] Nov 03 15:07:30.702176 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:07:30.702019 5060 cpu_manager.go:414] "containerMap: removing container" podUID="5383fccc-ffe8-48f9-9145-b4729cbe466f" containerName="revoke-keys" Nov 03 15:07:30.702176 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:07:30.702031 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5383fccc-ffe8-48f9-9145-b4729cbe466f" containerName="revoke-keys" Nov 03 15:07:30.702176 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:07:30.702049 5060 cpu_manager.go:414] "containerMap: removing container" podUID="b0d2f12e-9c6c-4e22-ba6f-197f411b0666" containerName="manager" Nov 03 15:07:30.702176 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:07:30.702054 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0d2f12e-9c6c-4e22-ba6f-197f411b0666" containerName="manager" Nov 03 15:07:30.702176 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:07:30.702131 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="5383fccc-ffe8-48f9-9145-b4729cbe466f" containerName="revoke-keys" Nov 03 15:07:30.702176 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:07:30.702141 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0d2f12e-9c6c-4e22-ba6f-197f411b0666" containerName="manager" Nov 03 15:07:30.705391 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:07:30.705373 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5rnng" Nov 03 15:07:30.723891 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:07:30.723873 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-47h52\"" Nov 03 15:07:30.750974 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:07:30.750945 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5rnng"] Nov 03 15:07:30.783191 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:07:30.783167 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz8qh\" (UniqueName: \"kubernetes.io/projected/af191281-ee95-447a-9713-4a922592fe53-kube-api-access-nz8qh\") pod \"kuadrant-operator-controller-manager-678d87766f-5rnng\" (UID: \"af191281-ee95-447a-9713-4a922592fe53\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5rnng" Nov 03 15:07:30.783315 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:07:30.783293 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/af191281-ee95-447a-9713-4a922592fe53-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-5rnng\" (UID: \"af191281-ee95-447a-9713-4a922592fe53\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5rnng" Nov 03 15:07:30.884718 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:07:30.884681 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz8qh\" (UniqueName: \"kubernetes.io/projected/af191281-ee95-447a-9713-4a922592fe53-kube-api-access-nz8qh\") pod \"kuadrant-operator-controller-manager-678d87766f-5rnng\" (UID: \"af191281-ee95-447a-9713-4a922592fe53\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5rnng" Nov 03 15:07:30.884922 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:07:30.884777 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/af191281-ee95-447a-9713-4a922592fe53-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-5rnng\" (UID: \"af191281-ee95-447a-9713-4a922592fe53\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5rnng" Nov 03 15:07:30.885172 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:07:30.885151 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/af191281-ee95-447a-9713-4a922592fe53-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-5rnng\" (UID: \"af191281-ee95-447a-9713-4a922592fe53\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5rnng" Nov 03 15:07:30.909019 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:07:30.908994 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz8qh\" (UniqueName: \"kubernetes.io/projected/af191281-ee95-447a-9713-4a922592fe53-kube-api-access-nz8qh\") pod \"kuadrant-operator-controller-manager-678d87766f-5rnng\" (UID: \"af191281-ee95-447a-9713-4a922592fe53\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5rnng" Nov 03 15:07:31.017524 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:07:31.017492 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5rnng" Nov 03 15:07:31.354889 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:07:31.354863 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5rnng"] Nov 03 15:07:31.357025 ip-10-0-143-171 kubenswrapper[5060]: W1103 15:07:31.356996 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf191281_ee95_447a_9713_4a922592fe53.slice/crio-82bc75f80f39ed01d715ea0fab3e350c263990a65b9f1b21c6c51a7699649329 WatchSource:0}: Error finding container 82bc75f80f39ed01d715ea0fab3e350c263990a65b9f1b21c6c51a7699649329: Status 404 returned error can't find the container with id 82bc75f80f39ed01d715ea0fab3e350c263990a65b9f1b21c6c51a7699649329 Nov 03 15:07:32.114141 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:07:32.114106 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5rnng" event={"ID":"af191281-ee95-447a-9713-4a922592fe53","Type":"ContainerStarted","Data":"58842d1f45655687a3abace26048d2257f5c69fddfc106df30d92e5bca73f78d"} Nov 03 15:07:32.114538 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:07:32.114164 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5rnng" event={"ID":"af191281-ee95-447a-9713-4a922592fe53","Type":"ContainerStarted","Data":"82bc75f80f39ed01d715ea0fab3e350c263990a65b9f1b21c6c51a7699649329"} Nov 03 15:07:32.114538 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:07:32.114188 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5rnng" Nov 03 15:07:32.188269 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:07:32.188221 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5rnng" podStartSLOduration=2.188207662 podStartE2EDuration="2.188207662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:07:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:07:32.173508553 +0000 UTC m=+3386.062298130" watchObservedRunningTime="2026-11-03 15:07:32.188207662 +0000 UTC m=+3386.076997243" Nov 03 15:07:39.438317 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:07:39.438280 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:07:43.120811 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:07:43.120781 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5rnng" Nov 03 15:08:09.454877 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:08:09.450042 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:08:25.303277 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:25.303237 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5757765976-nbj76"] Nov 03 15:08:25.306916 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:25.306894 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5757765976-nbj76" Nov 03 15:08:25.339112 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:25.339077 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 15:08:25.339370 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:25.339343 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 15:08:25.339370 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:25.339346 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 15:08:25.339993 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:25.339966 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 15:08:25.340665 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:25.340642 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7cpjn\"" Nov 03 15:08:25.350563 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:25.350523 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5757765976-nbj76"] Nov 03 15:08:25.451688 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:25.451656 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6691a038-03bd-4759-9e68-5a03aee7f9a4-openshift-service-ca\") pod \"maas-api-5757765976-nbj76\" (UID: \"6691a038-03bd-4759-9e68-5a03aee7f9a4\") " pod="odh-ai-gateway-infra/maas-api-5757765976-nbj76" Nov 03 15:08:25.451873 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:25.451714 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6691a038-03bd-4759-9e68-5a03aee7f9a4-maas-api-tls\") pod \"maas-api-5757765976-nbj76\" (UID: \"6691a038-03bd-4759-9e68-5a03aee7f9a4\") " pod="odh-ai-gateway-infra/maas-api-5757765976-nbj76" Nov 03 15:08:25.451873 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:25.451808 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6691a038-03bd-4759-9e68-5a03aee7f9a4-metrics-certs\") pod \"maas-api-5757765976-nbj76\" (UID: \"6691a038-03bd-4759-9e68-5a03aee7f9a4\") " pod="odh-ai-gateway-infra/maas-api-5757765976-nbj76" Nov 03 15:08:25.451952 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:25.451890 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mslj4\" (UniqueName: \"kubernetes.io/projected/6691a038-03bd-4759-9e68-5a03aee7f9a4-kube-api-access-mslj4\") pod \"maas-api-5757765976-nbj76\" (UID: \"6691a038-03bd-4759-9e68-5a03aee7f9a4\") " pod="odh-ai-gateway-infra/maas-api-5757765976-nbj76" Nov 03 15:08:25.552504 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:25.552466 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6691a038-03bd-4759-9e68-5a03aee7f9a4-metrics-certs\") pod \"maas-api-5757765976-nbj76\" (UID: \"6691a038-03bd-4759-9e68-5a03aee7f9a4\") " pod="odh-ai-gateway-infra/maas-api-5757765976-nbj76" Nov 03 15:08:25.552681 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:25.552517 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mslj4\" (UniqueName: \"kubernetes.io/projected/6691a038-03bd-4759-9e68-5a03aee7f9a4-kube-api-access-mslj4\") pod \"maas-api-5757765976-nbj76\" (UID: \"6691a038-03bd-4759-9e68-5a03aee7f9a4\") " pod="odh-ai-gateway-infra/maas-api-5757765976-nbj76" Nov 03 15:08:25.552681 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:25.552559 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6691a038-03bd-4759-9e68-5a03aee7f9a4-openshift-service-ca\") pod \"maas-api-5757765976-nbj76\" (UID: \"6691a038-03bd-4759-9e68-5a03aee7f9a4\") " pod="odh-ai-gateway-infra/maas-api-5757765976-nbj76" Nov 03 15:08:25.552681 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:25.552603 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6691a038-03bd-4759-9e68-5a03aee7f9a4-maas-api-tls\") pod \"maas-api-5757765976-nbj76\" (UID: \"6691a038-03bd-4759-9e68-5a03aee7f9a4\") " pod="odh-ai-gateway-infra/maas-api-5757765976-nbj76" Nov 03 15:08:25.553335 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:25.553279 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6691a038-03bd-4759-9e68-5a03aee7f9a4-openshift-service-ca\") pod \"maas-api-5757765976-nbj76\" (UID: \"6691a038-03bd-4759-9e68-5a03aee7f9a4\") " pod="odh-ai-gateway-infra/maas-api-5757765976-nbj76" Nov 03 15:08:25.554965 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:25.554936 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6691a038-03bd-4759-9e68-5a03aee7f9a4-metrics-certs\") pod \"maas-api-5757765976-nbj76\" (UID: \"6691a038-03bd-4759-9e68-5a03aee7f9a4\") " pod="odh-ai-gateway-infra/maas-api-5757765976-nbj76" Nov 03 15:08:25.555064 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:25.554962 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6691a038-03bd-4759-9e68-5a03aee7f9a4-maas-api-tls\") pod \"maas-api-5757765976-nbj76\" (UID: \"6691a038-03bd-4759-9e68-5a03aee7f9a4\") " pod="odh-ai-gateway-infra/maas-api-5757765976-nbj76" Nov 03 15:08:25.575300 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:25.575280 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mslj4\" (UniqueName: \"kubernetes.io/projected/6691a038-03bd-4759-9e68-5a03aee7f9a4-kube-api-access-mslj4\") pod \"maas-api-5757765976-nbj76\" (UID: \"6691a038-03bd-4759-9e68-5a03aee7f9a4\") " pod="odh-ai-gateway-infra/maas-api-5757765976-nbj76" Nov 03 15:08:25.619218 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:25.619196 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5757765976-nbj76" Nov 03 15:08:25.924680 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:25.924655 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5757765976-nbj76"] Nov 03 15:08:25.926850 ip-10-0-143-171 kubenswrapper[5060]: W1103 15:08:25.926811 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6691a038_03bd_4759_9e68_5a03aee7f9a4.slice/crio-2143452cdf0323683801095e585e42c1cff19aac91f44437d3c047cb2b600c15 WatchSource:0}: Error finding container 2143452cdf0323683801095e585e42c1cff19aac91f44437d3c047cb2b600c15: Status 404 returned error can't find the container with id 2143452cdf0323683801095e585e42c1cff19aac91f44437d3c047cb2b600c15 Nov 03 15:08:25.928178 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:25.928159 5060 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 15:08:26.439178 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:26.439141 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5757765976-nbj76" event={"ID":"6691a038-03bd-4759-9e68-5a03aee7f9a4","Type":"ContainerStarted","Data":"cfbc79cd6dc91e5daa5070a8aa25864862ed0abfce5b86394409cd92d655fed1"} Nov 03 15:08:26.439551 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:26.439198 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5757765976-nbj76" event={"ID":"6691a038-03bd-4759-9e68-5a03aee7f9a4","Type":"ContainerStarted","Data":"2143452cdf0323683801095e585e42c1cff19aac91f44437d3c047cb2b600c15"} Nov 03 15:08:26.439551 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:26.439229 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5757765976-nbj76" Nov 03 15:08:26.487011 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:26.486954 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5757765976-nbj76" podStartSLOduration=1.086115733 podStartE2EDuration="1.486929035s" totalImagesPullingTime="400.813302ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:08:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:08:26.486150083 +0000 UTC m=+3440.374939665" watchObservedRunningTime="2026-11-03 15:08:26.486929035 +0000 UTC m=+3440.375718619" Nov 03 15:08:32.450036 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:32.450005 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5757765976-nbj76" Nov 03 15:08:33.453190 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:33.453155 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-84b5dd6748-lx27t"] Nov 03 15:08:33.457290 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:33.457268 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84b5dd6748-lx27t" Nov 03 15:08:33.494261 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:33.494231 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84b5dd6748-lx27t"] Nov 03 15:08:33.625326 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:33.625295 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9qpj\" (UniqueName: \"kubernetes.io/projected/5a1d1b5e-f53f-40fb-9ec9-a3d40d0e8549-kube-api-access-g9qpj\") pod \"maas-api-84b5dd6748-lx27t\" (UID: \"5a1d1b5e-f53f-40fb-9ec9-a3d40d0e8549\") " pod="odh-ai-gateway-infra/maas-api-84b5dd6748-lx27t" Nov 03 15:08:33.625536 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:33.625336 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a1d1b5e-f53f-40fb-9ec9-a3d40d0e8549-metrics-certs\") pod \"maas-api-84b5dd6748-lx27t\" (UID: \"5a1d1b5e-f53f-40fb-9ec9-a3d40d0e8549\") " pod="odh-ai-gateway-infra/maas-api-84b5dd6748-lx27t" Nov 03 15:08:33.625536 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:33.625425 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a1d1b5e-f53f-40fb-9ec9-a3d40d0e8549-maas-api-tls\") pod \"maas-api-84b5dd6748-lx27t\" (UID: \"5a1d1b5e-f53f-40fb-9ec9-a3d40d0e8549\") " pod="odh-ai-gateway-infra/maas-api-84b5dd6748-lx27t" Nov 03 15:08:33.625536 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:33.625508 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a1d1b5e-f53f-40fb-9ec9-a3d40d0e8549-openshift-service-ca\") pod \"maas-api-84b5dd6748-lx27t\" (UID: \"5a1d1b5e-f53f-40fb-9ec9-a3d40d0e8549\") " pod="odh-ai-gateway-infra/maas-api-84b5dd6748-lx27t" Nov 03 15:08:33.726250 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:33.726167 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a1d1b5e-f53f-40fb-9ec9-a3d40d0e8549-metrics-certs\") pod \"maas-api-84b5dd6748-lx27t\" (UID: \"5a1d1b5e-f53f-40fb-9ec9-a3d40d0e8549\") " pod="odh-ai-gateway-infra/maas-api-84b5dd6748-lx27t" Nov 03 15:08:33.726250 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:33.726215 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a1d1b5e-f53f-40fb-9ec9-a3d40d0e8549-maas-api-tls\") pod \"maas-api-84b5dd6748-lx27t\" (UID: \"5a1d1b5e-f53f-40fb-9ec9-a3d40d0e8549\") " pod="odh-ai-gateway-infra/maas-api-84b5dd6748-lx27t" Nov 03 15:08:33.726487 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:33.726279 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a1d1b5e-f53f-40fb-9ec9-a3d40d0e8549-openshift-service-ca\") pod \"maas-api-84b5dd6748-lx27t\" (UID: \"5a1d1b5e-f53f-40fb-9ec9-a3d40d0e8549\") " pod="odh-ai-gateway-infra/maas-api-84b5dd6748-lx27t" Nov 03 15:08:33.726487 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:33.726309 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9qpj\" (UniqueName: \"kubernetes.io/projected/5a1d1b5e-f53f-40fb-9ec9-a3d40d0e8549-kube-api-access-g9qpj\") pod \"maas-api-84b5dd6748-lx27t\" (UID: \"5a1d1b5e-f53f-40fb-9ec9-a3d40d0e8549\") " pod="odh-ai-gateway-infra/maas-api-84b5dd6748-lx27t" Nov 03 15:08:33.727002 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:33.726973 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a1d1b5e-f53f-40fb-9ec9-a3d40d0e8549-openshift-service-ca\") pod \"maas-api-84b5dd6748-lx27t\" (UID: \"5a1d1b5e-f53f-40fb-9ec9-a3d40d0e8549\") " pod="odh-ai-gateway-infra/maas-api-84b5dd6748-lx27t" Nov 03 15:08:33.728699 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:33.728676 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a1d1b5e-f53f-40fb-9ec9-a3d40d0e8549-metrics-certs\") pod \"maas-api-84b5dd6748-lx27t\" (UID: \"5a1d1b5e-f53f-40fb-9ec9-a3d40d0e8549\") " pod="odh-ai-gateway-infra/maas-api-84b5dd6748-lx27t" Nov 03 15:08:33.728790 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:33.728724 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a1d1b5e-f53f-40fb-9ec9-a3d40d0e8549-maas-api-tls\") pod \"maas-api-84b5dd6748-lx27t\" (UID: \"5a1d1b5e-f53f-40fb-9ec9-a3d40d0e8549\") " pod="odh-ai-gateway-infra/maas-api-84b5dd6748-lx27t" Nov 03 15:08:33.752884 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:33.752852 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9qpj\" (UniqueName: \"kubernetes.io/projected/5a1d1b5e-f53f-40fb-9ec9-a3d40d0e8549-kube-api-access-g9qpj\") pod \"maas-api-84b5dd6748-lx27t\" (UID: \"5a1d1b5e-f53f-40fb-9ec9-a3d40d0e8549\") " pod="odh-ai-gateway-infra/maas-api-84b5dd6748-lx27t" Nov 03 15:08:33.770676 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:33.770654 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84b5dd6748-lx27t" Nov 03 15:08:34.084820 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:34.084791 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84b5dd6748-lx27t"] Nov 03 15:08:34.087778 ip-10-0-143-171 kubenswrapper[5060]: W1103 15:08:34.087748 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a1d1b5e_f53f_40fb_9ec9_a3d40d0e8549.slice/crio-049777b33b4c5d65ed21d14d041f19a30f9488fd7a4731865544e6900b8f3644 WatchSource:0}: Error finding container 049777b33b4c5d65ed21d14d041f19a30f9488fd7a4731865544e6900b8f3644: Status 404 returned error can't find the container with id 049777b33b4c5d65ed21d14d041f19a30f9488fd7a4731865544e6900b8f3644 Nov 03 15:08:34.480820 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:34.480781 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84b5dd6748-lx27t" event={"ID":"5a1d1b5e-f53f-40fb-9ec9-a3d40d0e8549","Type":"ContainerStarted","Data":"dc3a09c8764d3a9210fe94fb6b27fa7a29066d1c06930d25947f496a03d73942"} Nov 03 15:08:34.481255 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:34.480866 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84b5dd6748-lx27t" event={"ID":"5a1d1b5e-f53f-40fb-9ec9-a3d40d0e8549","Type":"ContainerStarted","Data":"049777b33b4c5d65ed21d14d041f19a30f9488fd7a4731865544e6900b8f3644"} Nov 03 15:08:34.481255 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:34.480898 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-84b5dd6748-lx27t" Nov 03 15:08:34.533640 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:34.533590 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-84b5dd6748-lx27t" podStartSLOduration=1.2242343 podStartE2EDuration="1.533576686s" totalImagesPullingTime="309.342386ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:08:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:08:34.53155661 +0000 UTC m=+3448.420346193" watchObservedRunningTime="2026-11-03 15:08:34.533576686 +0000 UTC m=+3448.422366266" Nov 03 15:08:39.439105 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:08:39.439064 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:08:40.492247 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:40.492217 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-84b5dd6748-lx27t" Nov 03 15:08:40.577329 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:40.577296 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5757765976-nbj76"] Nov 03 15:08:40.577560 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:40.577528 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5757765976-nbj76" podUID="6691a038-03bd-4759-9e68-5a03aee7f9a4" containerName="maas-api" containerID="cri-o://cfbc79cd6dc91e5daa5070a8aa25864862ed0abfce5b86394409cd92d655fed1" gracePeriod=30 Nov 03 15:08:40.832509 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:40.832475 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5757765976-nbj76" Nov 03 15:08:40.909217 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:40.909186 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6691a038-03bd-4759-9e68-5a03aee7f9a4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6691a038-03bd-4759-9e68-5a03aee7f9a4-openshift-service-ca\") pod \"6691a038-03bd-4759-9e68-5a03aee7f9a4\" (UID: \"6691a038-03bd-4759-9e68-5a03aee7f9a4\") " Nov 03 15:08:40.909377 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:40.909235 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6691a038-03bd-4759-9e68-5a03aee7f9a4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6691a038-03bd-4759-9e68-5a03aee7f9a4-maas-api-tls\") pod \"6691a038-03bd-4759-9e68-5a03aee7f9a4\" (UID: \"6691a038-03bd-4759-9e68-5a03aee7f9a4\") " Nov 03 15:08:40.909377 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:40.909289 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6691a038-03bd-4759-9e68-5a03aee7f9a4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6691a038-03bd-4759-9e68-5a03aee7f9a4-metrics-certs\") pod \"6691a038-03bd-4759-9e68-5a03aee7f9a4\" (UID: \"6691a038-03bd-4759-9e68-5a03aee7f9a4\") " Nov 03 15:08:40.909518 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:40.909422 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6691a038-03bd-4759-9e68-5a03aee7f9a4-kube-api-access-mslj4\" (UniqueName: \"kubernetes.io/projected/6691a038-03bd-4759-9e68-5a03aee7f9a4-kube-api-access-mslj4\") pod \"6691a038-03bd-4759-9e68-5a03aee7f9a4\" (UID: \"6691a038-03bd-4759-9e68-5a03aee7f9a4\") " Nov 03 15:08:40.909640 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:40.909616 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6691a038-03bd-4759-9e68-5a03aee7f9a4-openshift-service-ca" pod "6691a038-03bd-4759-9e68-5a03aee7f9a4" (UID: "6691a038-03bd-4759-9e68-5a03aee7f9a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:08:40.909875 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:40.909825 5060 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6691a038-03bd-4759-9e68-5a03aee7f9a4-openshift-service-ca\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 15:08:40.911484 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:40.911454 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6691a038-03bd-4759-9e68-5a03aee7f9a4-maas-api-tls" pod "6691a038-03bd-4759-9e68-5a03aee7f9a4" (UID: "6691a038-03bd-4759-9e68-5a03aee7f9a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:08:40.911588 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:40.911524 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6691a038-03bd-4759-9e68-5a03aee7f9a4-metrics-certs" pod "6691a038-03bd-4759-9e68-5a03aee7f9a4" (UID: "6691a038-03bd-4759-9e68-5a03aee7f9a4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:08:40.911588 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:40.911563 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6691a038-03bd-4759-9e68-5a03aee7f9a4-kube-api-access-mslj4" pod "6691a038-03bd-4759-9e68-5a03aee7f9a4" (UID: "6691a038-03bd-4759-9e68-5a03aee7f9a4"). InnerVolumeSpecName "kube-api-access-mslj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:08:41.010455 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:41.010383 5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6691a038-03bd-4759-9e68-5a03aee7f9a4-maas-api-tls\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 15:08:41.010455 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:41.010408 5060 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6691a038-03bd-4759-9e68-5a03aee7f9a4-metrics-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 15:08:41.010455 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:41.010419 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mslj4\" (UniqueName: \"kubernetes.io/projected/6691a038-03bd-4759-9e68-5a03aee7f9a4-kube-api-access-mslj4\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 15:08:41.535061 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:41.535026 5060 generic.go:356] "Generic (PLEG): container finished" podID="6691a038-03bd-4759-9e68-5a03aee7f9a4" containerID="cfbc79cd6dc91e5daa5070a8aa25864862ed0abfce5b86394409cd92d655fed1" exitCode=0 Nov 03 15:08:41.535480 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:41.535075 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5757765976-nbj76" Nov 03 15:08:41.535480 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:41.535092 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5757765976-nbj76" event={"ID":"6691a038-03bd-4759-9e68-5a03aee7f9a4","Type":"ContainerDied","Data":"cfbc79cd6dc91e5daa5070a8aa25864862ed0abfce5b86394409cd92d655fed1"} Nov 03 15:08:41.535480 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:41.535142 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5757765976-nbj76" event={"ID":"6691a038-03bd-4759-9e68-5a03aee7f9a4","Type":"ContainerDied","Data":"2143452cdf0323683801095e585e42c1cff19aac91f44437d3c047cb2b600c15"} Nov 03 15:08:41.535480 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:41.535171 5060 scope.go:122] "RemoveContainer" containerID="cfbc79cd6dc91e5daa5070a8aa25864862ed0abfce5b86394409cd92d655fed1" Nov 03 15:08:41.544622 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:41.544603 5060 scope.go:122] "RemoveContainer" containerID="cfbc79cd6dc91e5daa5070a8aa25864862ed0abfce5b86394409cd92d655fed1" Nov 03 15:08:41.544895 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:08:41.544875 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfbc79cd6dc91e5daa5070a8aa25864862ed0abfce5b86394409cd92d655fed1\": container with ID starting with cfbc79cd6dc91e5daa5070a8aa25864862ed0abfce5b86394409cd92d655fed1 not found: ID does not exist" containerID="cfbc79cd6dc91e5daa5070a8aa25864862ed0abfce5b86394409cd92d655fed1" Nov 03 15:08:41.544948 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:41.544907 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfbc79cd6dc91e5daa5070a8aa25864862ed0abfce5b86394409cd92d655fed1"} err="failed to get container status \"cfbc79cd6dc91e5daa5070a8aa25864862ed0abfce5b86394409cd92d655fed1\": rpc error: code = NotFound desc = could not find container \"cfbc79cd6dc91e5daa5070a8aa25864862ed0abfce5b86394409cd92d655fed1\": container with ID starting with cfbc79cd6dc91e5daa5070a8aa25864862ed0abfce5b86394409cd92d655fed1 not found: ID does not exist" Nov 03 15:08:41.592396 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:41.592365 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5757765976-nbj76"] Nov 03 15:08:41.605088 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:41.605060 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5757765976-nbj76"] Nov 03 15:08:42.247745 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:08:42.247713 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6691a038-03bd-4759-9e68-5a03aee7f9a4" path="/var/lib/kubelet/pods/6691a038-03bd-4759-9e68-5a03aee7f9a4/volumes" Nov 03 15:09:09.438594 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:09:09.438551 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:09:39.438299 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:09:39.438264 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:10:09.438379 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:10:09.438331 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:10:39.439641 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:10:39.439602 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:11:07.647427 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:11:07.647299 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/3.log" Nov 03 15:11:07.653600 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:11:07.653578 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/3.log" Nov 03 15:11:09.438856 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:11:09.438799 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:11:39.438676 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:11:39.438640 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:12:09.438277 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:12:09.438245 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:12:39.438688 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:12:39.438650 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:13:09.438351 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:13:09.438313 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:13:39.439202 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:13:39.439162 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:14:09.439259 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:14:09.439224 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:14:13.447404 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:13.447362 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9"] Nov 03 15:14:13.447924 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:13.447682 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" podUID="3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52" containerName="istio-proxy" containerID="cri-o://7f37cccf66f94ab7c4ab94e9c201318739263d25f74b25516a55295883937a57" gracePeriod=30 Nov 03 15:14:13.460041 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:13.460005 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm"] Nov 03 15:14:13.460250 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:13.460209 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" podUID="796dbd66-d335-4eed-bb70-073c3971adae" containerName="istio-proxy" containerID="cri-o://6a8a3b3c576c0f6a2d832c56f87cbc94d5946050f19b61eecd717a9a24629c00" gracePeriod=30 Nov 03 15:14:13.462289 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:13.462265 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv"] Nov 03 15:14:13.462441 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:13.462422 5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" podUID="a88f6aee-6ba7-4821-8da3-c86c4c56dcd1" containerName="istio-proxy" containerID="cri-o://9f0a2cd5dfc0a1084149057b9ed160f146a706d081263f16e2591a48abf62b59" gracePeriod=30 Nov 03 15:14:18.730522 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.730488 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" Nov 03 15:14:18.755003 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.754972 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-istio-data\") pod \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\" (UID: \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\") " Nov 03 15:14:18.755003 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.755001 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-istio-podinfo\") pod \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\" (UID: \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\") " Nov 03 15:14:18.755190 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.755023 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-istiod-ca-cert\") pod \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\" (UID: \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\") " Nov 03 15:14:18.755190 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.755090 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-credential-socket\") pod \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\" (UID: \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\") " Nov 03 15:14:18.755190 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.755152 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-workload-certs\") pod \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\" (UID: \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\") " Nov 03 15:14:18.755357 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.755217 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-istio-token\" (UniqueName: \"kubernetes.io/projected/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-istio-token\") pod \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\" (UID: \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\") " Nov 03 15:14:18.755357 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.755266 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-istio-data" pod "3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52" (UID: "3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:14:18.755357 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.755290 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-kube-api-access-ptt76\" (UniqueName: \"kubernetes.io/projected/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-kube-api-access-ptt76\") pod \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\" (UID: \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\") " Nov 03 15:14:18.755512 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.755466 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-workload-certs" pod "3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52" (UID: "3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:14:18.755632 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.755609 5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-istio-data\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.755711 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.755633 5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-workload-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.755845 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.755806 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-credential-socket" pod "3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52" (UID: "3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:14:18.755985 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.755958 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-istiod-ca-cert" pod "3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52" (UID: "3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:14:18.757640 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.757615 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-kube-api-access-ptt76" pod "3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52" (UID: "3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52"). InnerVolumeSpecName "kube-api-access-ptt76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:14:18.757640 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.757627 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-istio-token" pod "3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52" (UID: "3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:14:18.757978 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.757957 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-istio-podinfo" pod "3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52" (UID: "3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:14:18.772521 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.772499 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" Nov 03 15:14:18.774549 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.774533 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" Nov 03 15:14:18.855941 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.855904 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/796dbd66-d335-4eed-bb70-073c3971adae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/796dbd66-d335-4eed-bb70-073c3971adae-istiod-ca-cert\") pod \"796dbd66-d335-4eed-bb70-073c3971adae\" (UID: \"796dbd66-d335-4eed-bb70-073c3971adae\") " Nov 03 15:14:18.856136 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.855963 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/796dbd66-d335-4eed-bb70-073c3971adae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/796dbd66-d335-4eed-bb70-073c3971adae-credential-socket\") pod \"796dbd66-d335-4eed-bb70-073c3971adae\" (UID: \"796dbd66-d335-4eed-bb70-073c3971adae\") " Nov 03 15:14:18.856136 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.856019 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/796dbd66-d335-4eed-bb70-073c3971adae-istio-token\" (UniqueName: \"kubernetes.io/projected/796dbd66-d335-4eed-bb70-073c3971adae-istio-token\") pod \"796dbd66-d335-4eed-bb70-073c3971adae\" (UID: \"796dbd66-d335-4eed-bb70-073c3971adae\") " Nov 03 15:14:18.856136 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.856056 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-istio-token\" (UniqueName: \"kubernetes.io/projected/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-istio-token\") pod \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\" (UID: \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\") " Nov 03 15:14:18.856136 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.856083 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/796dbd66-d335-4eed-bb70-073c3971adae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/796dbd66-d335-4eed-bb70-073c3971adae-istio-envoy\") pod \"796dbd66-d335-4eed-bb70-073c3971adae\" (UID: \"796dbd66-d335-4eed-bb70-073c3971adae\") " Nov 03 15:14:18.856136 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.856108 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-istiod-ca-cert\") pod \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\" (UID: \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\") " Nov 03 15:14:18.856416 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.856138 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-istio-podinfo\") pod \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\" (UID: \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\") " Nov 03 15:14:18.856416 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.856167 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-workload-socket\") pod \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\" (UID: \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\") " Nov 03 15:14:18.856416 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.856195 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-credential-socket\") pod \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\" (UID: \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\") " Nov 03 15:14:18.856416 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.856221 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-istio-envoy\") pod \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\" (UID: \"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52\") " Nov 03 15:14:18.856416 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.856245 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/796dbd66-d335-4eed-bb70-073c3971adae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/796dbd66-d335-4eed-bb70-073c3971adae-istio-podinfo\") pod \"796dbd66-d335-4eed-bb70-073c3971adae\" (UID: \"796dbd66-d335-4eed-bb70-073c3971adae\") " Nov 03 15:14:18.856416 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.856289 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-istio-data\") pod \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\" (UID: \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\") " Nov 03 15:14:18.856416 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.856319 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-istio-envoy\") pod \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\" (UID: \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\") " Nov 03 15:14:18.856416 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.856356 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-kube-api-access-g8ldx\" (UniqueName: \"kubernetes.io/projected/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-kube-api-access-g8ldx\") pod \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\" (UID: \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\") " Nov 03 15:14:18.856416 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.856380 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-workload-certs\") pod \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\" (UID: \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\") " Nov 03 15:14:18.856416 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.856405 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/796dbd66-d335-4eed-bb70-073c3971adae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/796dbd66-d335-4eed-bb70-073c3971adae-workload-certs\") pod \"796dbd66-d335-4eed-bb70-073c3971adae\" (UID: \"796dbd66-d335-4eed-bb70-073c3971adae\") " Nov 03 15:14:18.856942 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.856427 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/796dbd66-d335-4eed-bb70-073c3971adae-kube-api-access-xwjb6\" (UniqueName: \"kubernetes.io/projected/796dbd66-d335-4eed-bb70-073c3971adae-kube-api-access-xwjb6\") pod \"796dbd66-d335-4eed-bb70-073c3971adae\" (UID: \"796dbd66-d335-4eed-bb70-073c3971adae\") " Nov 03 15:14:18.856942 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.856470 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-workload-socket\") pod \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\" (UID: \"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1\") " Nov 03 15:14:18.856942 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.856498 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/796dbd66-d335-4eed-bb70-073c3971adae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/796dbd66-d335-4eed-bb70-073c3971adae-istio-data\") pod \"796dbd66-d335-4eed-bb70-073c3971adae\" (UID: \"796dbd66-d335-4eed-bb70-073c3971adae\") " Nov 03 15:14:18.856942 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.856559 5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/796dbd66-d335-4eed-bb70-073c3971adae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/796dbd66-d335-4eed-bb70-073c3971adae-workload-socket\") pod \"796dbd66-d335-4eed-bb70-073c3971adae\" (UID: \"796dbd66-d335-4eed-bb70-073c3971adae\") " Nov 03 15:14:18.856942 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.856759 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-credential-socket" pod "a88f6aee-6ba7-4821-8da3-c86c4c56dcd1" (UID: "a88f6aee-6ba7-4821-8da3-c86c4c56dcd1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:14:18.856942 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.856916 5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-istio-token\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.856942 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.856937 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptt76\" (UniqueName: \"kubernetes.io/projected/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-kube-api-access-ptt76\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.857288 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.856954 5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-credential-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.857288 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.856999 5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-istio-podinfo\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.857288 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.857013 5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-istiod-ca-cert\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.857288 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.857019 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/796dbd66-d335-4eed-bb70-073c3971adae-workload-socket" pod "796dbd66-d335-4eed-bb70-073c3971adae" (UID: "796dbd66-d335-4eed-bb70-073c3971adae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:14:18.857288 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.857028 5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-credential-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.857543 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.857389 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/796dbd66-d335-4eed-bb70-073c3971adae-istiod-ca-cert" pod "796dbd66-d335-4eed-bb70-073c3971adae" (UID: "796dbd66-d335-4eed-bb70-073c3971adae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:14:18.857706 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.857626 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-workload-socket" pod "3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52" (UID: "3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:14:18.858381 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.858109 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-workload-socket" pod "a88f6aee-6ba7-4821-8da3-c86c4c56dcd1" (UID: "a88f6aee-6ba7-4821-8da3-c86c4c56dcd1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:14:18.858381 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.858240 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-istio-data" pod "a88f6aee-6ba7-4821-8da3-c86c4c56dcd1" (UID: "a88f6aee-6ba7-4821-8da3-c86c4c56dcd1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:14:18.858648 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.858623 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-istiod-ca-cert" pod "a88f6aee-6ba7-4821-8da3-c86c4c56dcd1" (UID: "a88f6aee-6ba7-4821-8da3-c86c4c56dcd1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:14:18.858769 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.858670 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-workload-certs" pod "a88f6aee-6ba7-4821-8da3-c86c4c56dcd1" (UID: "a88f6aee-6ba7-4821-8da3-c86c4c56dcd1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:14:18.859197 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.858950 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/796dbd66-d335-4eed-bb70-073c3971adae-credential-socket" pod "796dbd66-d335-4eed-bb70-073c3971adae" (UID: "796dbd66-d335-4eed-bb70-073c3971adae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:14:18.859197 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.859143 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/796dbd66-d335-4eed-bb70-073c3971adae-istio-data" pod "796dbd66-d335-4eed-bb70-073c3971adae" (UID: "796dbd66-d335-4eed-bb70-073c3971adae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:14:18.859364 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.859344 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/796dbd66-d335-4eed-bb70-073c3971adae-workload-certs" pod "796dbd66-d335-4eed-bb70-073c3971adae" (UID: "796dbd66-d335-4eed-bb70-073c3971adae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:14:18.859994 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.859652 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/796dbd66-d335-4eed-bb70-073c3971adae-istio-envoy" pod "796dbd66-d335-4eed-bb70-073c3971adae" (UID: "796dbd66-d335-4eed-bb70-073c3971adae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:14:18.863009 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.862981 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/796dbd66-d335-4eed-bb70-073c3971adae-kube-api-access-xwjb6" pod "796dbd66-d335-4eed-bb70-073c3971adae" (UID: "796dbd66-d335-4eed-bb70-073c3971adae"). InnerVolumeSpecName "kube-api-access-xwjb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:14:18.863118 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.863018 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/796dbd66-d335-4eed-bb70-073c3971adae-istio-token" pod "796dbd66-d335-4eed-bb70-073c3971adae" (UID: "796dbd66-d335-4eed-bb70-073c3971adae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:14:18.863188 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.863136 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-kube-api-access-g8ldx" pod "a88f6aee-6ba7-4821-8da3-c86c4c56dcd1" (UID: "a88f6aee-6ba7-4821-8da3-c86c4c56dcd1"). InnerVolumeSpecName "kube-api-access-g8ldx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:14:18.863251 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.863183 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/796dbd66-d335-4eed-bb70-073c3971adae-istio-podinfo" pod "796dbd66-d335-4eed-bb70-073c3971adae" (UID: "796dbd66-d335-4eed-bb70-073c3971adae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:14:18.863455 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.863427 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-istio-envoy" pod "a88f6aee-6ba7-4821-8da3-c86c4c56dcd1" (UID: "a88f6aee-6ba7-4821-8da3-c86c4c56dcd1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:14:18.864035 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.864012 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-istio-envoy" pod "3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52" (UID: "3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 15:14:18.864353 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.864330 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-istio-token" pod "a88f6aee-6ba7-4821-8da3-c86c4c56dcd1" (UID: "a88f6aee-6ba7-4821-8da3-c86c4c56dcd1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:14:18.864443 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.864426 5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-istio-podinfo" pod "a88f6aee-6ba7-4821-8da3-c86c4c56dcd1" (UID: "a88f6aee-6ba7-4821-8da3-c86c4c56dcd1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 15:14:18.957966 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.957926 5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-workload-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.957966 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.957965 5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/796dbd66-d335-4eed-bb70-073c3971adae-istio-data\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.958168 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.957981 5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/796dbd66-d335-4eed-bb70-073c3971adae-workload-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.958168 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.957995 5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/796dbd66-d335-4eed-bb70-073c3971adae-istiod-ca-cert\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.958168 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.958014 5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/796dbd66-d335-4eed-bb70-073c3971adae-credential-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.958168 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.958027 5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/796dbd66-d335-4eed-bb70-073c3971adae-istio-token\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.958168 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.958040 5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-istio-token\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.958168 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.958053 5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/796dbd66-d335-4eed-bb70-073c3971adae-istio-envoy\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.958168 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.958065 5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-istiod-ca-cert\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.958168 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.958078 5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-istio-podinfo\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.958168 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.958092 5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-workload-socket\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.958168 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.958106 5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52-istio-envoy\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.958168 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.958119 5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/796dbd66-d335-4eed-bb70-073c3971adae-istio-podinfo\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.958168 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.958133 5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-istio-data\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.958168 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.958146 5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-istio-envoy\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.958168 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.958159 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8ldx\" (UniqueName: \"kubernetes.io/projected/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-kube-api-access-g8ldx\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.958625 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.958173 5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1-workload-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.958625 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.958188 5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/796dbd66-d335-4eed-bb70-073c3971adae-workload-certs\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 15:14:18.958625 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:18.958201 5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwjb6\" (UniqueName: \"kubernetes.io/projected/796dbd66-d335-4eed-bb70-073c3971adae-kube-api-access-xwjb6\") on node \"ip-10-0-143-171.ec2.internal\" DevicePath \"\"" Nov 03 15:14:19.269981 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:19.269812 5060 generic.go:356] "Generic (PLEG): container finished" podID="a88f6aee-6ba7-4821-8da3-c86c4c56dcd1" containerID="9f0a2cd5dfc0a1084149057b9ed160f146a706d081263f16e2591a48abf62b59" exitCode=0 Nov 03 15:14:19.269981 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:19.269909 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" event={"ID":"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1","Type":"ContainerDied","Data":"9f0a2cd5dfc0a1084149057b9ed160f146a706d081263f16e2591a48abf62b59"} Nov 03 15:14:19.269981 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:19.269935 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" Nov 03 15:14:19.269981 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:19.269963 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv" event={"ID":"a88f6aee-6ba7-4821-8da3-c86c4c56dcd1","Type":"ContainerDied","Data":"a5173a90112f77d9b32e0e12ce96ca587033816a1e22a3dfd8a5f06b8d01e194"} Nov 03 15:14:19.270307 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:19.269992 5060 scope.go:122] "RemoveContainer" containerID="9f0a2cd5dfc0a1084149057b9ed160f146a706d081263f16e2591a48abf62b59" Nov 03 15:14:19.271540 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:19.271481 5060 generic.go:356] "Generic (PLEG): container finished" podID="796dbd66-d335-4eed-bb70-073c3971adae" containerID="6a8a3b3c576c0f6a2d832c56f87cbc94d5946050f19b61eecd717a9a24629c00" exitCode=0 Nov 03 15:14:19.271679 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:19.271549 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" Nov 03 15:14:19.271679 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:19.271563 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" event={"ID":"796dbd66-d335-4eed-bb70-073c3971adae","Type":"ContainerDied","Data":"6a8a3b3c576c0f6a2d832c56f87cbc94d5946050f19b61eecd717a9a24629c00"} Nov 03 15:14:19.271679 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:19.271609 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm" event={"ID":"796dbd66-d335-4eed-bb70-073c3971adae","Type":"ContainerDied","Data":"15dd71d9a731d6dd357bfa223aeaa78f959184004b55f9c36c629a807b1e0d6c"} Nov 03 15:14:19.273204 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:19.273176 5060 generic.go:356] "Generic (PLEG): container finished" podID="3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52" containerID="7f37cccf66f94ab7c4ab94e9c201318739263d25f74b25516a55295883937a57" exitCode=0 Nov 03 15:14:19.273334 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:19.273230 5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" Nov 03 15:14:19.273334 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:19.273250 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" event={"ID":"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52","Type":"ContainerDied","Data":"7f37cccf66f94ab7c4ab94e9c201318739263d25f74b25516a55295883937a57"} Nov 03 15:14:19.273334 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:19.273293 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9" event={"ID":"3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52","Type":"ContainerDied","Data":"17ec538eb3886c023ed2fd568c3efb538d41ce9705f36a1d87411235969e0f90"} Nov 03 15:14:19.281417 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:19.281397 5060 scope.go:122] "RemoveContainer" containerID="9f0a2cd5dfc0a1084149057b9ed160f146a706d081263f16e2591a48abf62b59" Nov 03 15:14:19.281749 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:14:19.281720 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f0a2cd5dfc0a1084149057b9ed160f146a706d081263f16e2591a48abf62b59\": container with ID starting with 9f0a2cd5dfc0a1084149057b9ed160f146a706d081263f16e2591a48abf62b59 not found: ID does not exist" containerID="9f0a2cd5dfc0a1084149057b9ed160f146a706d081263f16e2591a48abf62b59" Nov 03 15:14:19.281869 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:19.281749 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f0a2cd5dfc0a1084149057b9ed160f146a706d081263f16e2591a48abf62b59"} err="failed to get container status \"9f0a2cd5dfc0a1084149057b9ed160f146a706d081263f16e2591a48abf62b59\": rpc error: code = NotFound desc = could not find container \"9f0a2cd5dfc0a1084149057b9ed160f146a706d081263f16e2591a48abf62b59\": container with ID starting with 9f0a2cd5dfc0a1084149057b9ed160f146a706d081263f16e2591a48abf62b59 not found: ID does not exist" Nov 03 15:14:19.281869 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:19.281772 5060 scope.go:122] "RemoveContainer" containerID="6a8a3b3c576c0f6a2d832c56f87cbc94d5946050f19b61eecd717a9a24629c00" Nov 03 15:14:19.290951 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:19.290932 5060 scope.go:122] "RemoveContainer" containerID="6a8a3b3c576c0f6a2d832c56f87cbc94d5946050f19b61eecd717a9a24629c00" Nov 03 15:14:19.291178 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:14:19.291162 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a8a3b3c576c0f6a2d832c56f87cbc94d5946050f19b61eecd717a9a24629c00\": container with ID starting with 6a8a3b3c576c0f6a2d832c56f87cbc94d5946050f19b61eecd717a9a24629c00 not found: ID does not exist" containerID="6a8a3b3c576c0f6a2d832c56f87cbc94d5946050f19b61eecd717a9a24629c00" Nov 03 15:14:19.291236 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:19.291183 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a8a3b3c576c0f6a2d832c56f87cbc94d5946050f19b61eecd717a9a24629c00"} err="failed to get container status \"6a8a3b3c576c0f6a2d832c56f87cbc94d5946050f19b61eecd717a9a24629c00\": rpc error: code = NotFound desc = could not find container \"6a8a3b3c576c0f6a2d832c56f87cbc94d5946050f19b61eecd717a9a24629c00\": container with ID starting with 6a8a3b3c576c0f6a2d832c56f87cbc94d5946050f19b61eecd717a9a24629c00 not found: ID does not exist" Nov 03 15:14:19.291236 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:19.291199 5060 scope.go:122] "RemoveContainer" containerID="7f37cccf66f94ab7c4ab94e9c201318739263d25f74b25516a55295883937a57" Nov 03 15:14:19.299163 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:19.299145 5060 scope.go:122] "RemoveContainer" containerID="7f37cccf66f94ab7c4ab94e9c201318739263d25f74b25516a55295883937a57" Nov 03 15:14:19.299403 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:14:19.299386 5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f37cccf66f94ab7c4ab94e9c201318739263d25f74b25516a55295883937a57\": container with ID starting with 7f37cccf66f94ab7c4ab94e9c201318739263d25f74b25516a55295883937a57 not found: ID does not exist" containerID="7f37cccf66f94ab7c4ab94e9c201318739263d25f74b25516a55295883937a57" Nov 03 15:14:19.299454 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:19.299410 5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f37cccf66f94ab7c4ab94e9c201318739263d25f74b25516a55295883937a57"} err="failed to get container status \"7f37cccf66f94ab7c4ab94e9c201318739263d25f74b25516a55295883937a57\": rpc error: code = NotFound desc = could not find container \"7f37cccf66f94ab7c4ab94e9c201318739263d25f74b25516a55295883937a57\": container with ID starting with 7f37cccf66f94ab7c4ab94e9c201318739263d25f74b25516a55295883937a57 not found: ID does not exist" Nov 03 15:14:19.331224 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:19.331199 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm"] Nov 03 15:14:19.344203 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:19.344179 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9qpkm"] Nov 03 15:14:19.385204 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:19.385178 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv"] Nov 03 15:14:19.394097 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:19.394073 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7x8hv"] Nov 03 15:14:19.436681 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:19.436656 5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9"] Nov 03 15:14:19.448683 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:19.448654 5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8r2q9"] Nov 03 15:14:20.248311 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:20.248268 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52" path="/var/lib/kubelet/pods/3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52/volumes" Nov 03 15:14:20.248736 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:20.248721 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="796dbd66-d335-4eed-bb70-073c3971adae" path="/var/lib/kubelet/pods/796dbd66-d335-4eed-bb70-073c3971adae/volumes" Nov 03 15:14:20.249151 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:14:20.249138 5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a88f6aee-6ba7-4821-8da3-c86c4c56dcd1" path="/var/lib/kubelet/pods/a88f6aee-6ba7-4821-8da3-c86c4c56dcd1/volumes" Nov 03 15:14:39.437854 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:14:39.437809 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:15:09.438522 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:15:09.438479 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:15:39.439049 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:15:39.439004 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:16:07.678888 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:07.678763 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/3.log" Nov 03 15:16:07.693484 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:07.693456 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/3.log" Nov 03 15:16:09.438973 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:16:09.438936 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:16:13.829666 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:13.829631 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9"] Nov 03 15:16:13.830159 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:13.830031 5060 cpu_manager.go:414] "containerMap: removing container" podUID="3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52" containerName="istio-proxy" Nov 03 15:16:13.830159 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:13.830087 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52" containerName="istio-proxy" Nov 03 15:16:13.830159 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:13.830106 5060 cpu_manager.go:414] "containerMap: removing container" podUID="796dbd66-d335-4eed-bb70-073c3971adae" containerName="istio-proxy" Nov 03 15:16:13.830159 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:13.830112 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="796dbd66-d335-4eed-bb70-073c3971adae" containerName="istio-proxy" Nov 03 15:16:13.830159 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:13.830121 5060 cpu_manager.go:414] "containerMap: removing container" podUID="6691a038-03bd-4759-9e68-5a03aee7f9a4" containerName="maas-api" Nov 03 15:16:13.830159 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:13.830126 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6691a038-03bd-4759-9e68-5a03aee7f9a4" containerName="maas-api" Nov 03 15:16:13.830159 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:13.830141 5060 cpu_manager.go:414] "containerMap: removing container" podUID="a88f6aee-6ba7-4821-8da3-c86c4c56dcd1" containerName="istio-proxy" Nov 03 15:16:13.830159 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:13.830147 5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a88f6aee-6ba7-4821-8da3-c86c4c56dcd1" containerName="istio-proxy" Nov 03 15:16:13.830480 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:13.830211 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a4e5c0d-efd0-4862-91a9-7a9ba60f3c52" containerName="istio-proxy" Nov 03 15:16:13.830480 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:13.830222 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="796dbd66-d335-4eed-bb70-073c3971adae" containerName="istio-proxy" Nov 03 15:16:13.830480 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:13.830228 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="6691a038-03bd-4759-9e68-5a03aee7f9a4" containerName="maas-api" Nov 03 15:16:13.830480 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:13.830237 5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a88f6aee-6ba7-4821-8da3-c86c4c56dcd1" containerName="istio-proxy" Nov 03 15:16:13.833415 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:13.833398 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" Nov 03 15:16:13.855374 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:13.855349 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 15:16:13.866395 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:13.866368 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9"] Nov 03 15:16:13.937212 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:13.937182 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4nspm\"" Nov 03 15:16:13.955552 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:13.955522 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vczq8\" (UniqueName: \"kubernetes.io/projected/aaa56c30-5196-48ae-b337-220703d6809f-kube-api-access-vczq8\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t7kz9\" (UID: \"aaa56c30-5196-48ae-b337-220703d6809f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" Nov 03 15:16:13.955652 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:13.955563 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaa56c30-5196-48ae-b337-220703d6809f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t7kz9\" (UID: \"aaa56c30-5196-48ae-b337-220703d6809f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" Nov 03 15:16:13.955652 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:13.955589 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaa56c30-5196-48ae-b337-220703d6809f-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t7kz9\" (UID: \"aaa56c30-5196-48ae-b337-220703d6809f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" Nov 03 15:16:13.955728 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:13.955646 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaa56c30-5196-48ae-b337-220703d6809f-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t7kz9\" (UID: \"aaa56c30-5196-48ae-b337-220703d6809f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" Nov 03 15:16:13.955728 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:13.955697 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaa56c30-5196-48ae-b337-220703d6809f-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t7kz9\" (UID: \"aaa56c30-5196-48ae-b337-220703d6809f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" Nov 03 15:16:13.955792 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:13.955740 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaa56c30-5196-48ae-b337-220703d6809f-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t7kz9\" (UID: \"aaa56c30-5196-48ae-b337-220703d6809f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" Nov 03 15:16:13.955792 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:13.955755 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaa56c30-5196-48ae-b337-220703d6809f-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t7kz9\" (UID: \"aaa56c30-5196-48ae-b337-220703d6809f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" Nov 03 15:16:13.955792 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:13.955775 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaa56c30-5196-48ae-b337-220703d6809f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t7kz9\" (UID: \"aaa56c30-5196-48ae-b337-220703d6809f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" Nov 03 15:16:13.955963 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:13.955946 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaa56c30-5196-48ae-b337-220703d6809f-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t7kz9\" (UID: \"aaa56c30-5196-48ae-b337-220703d6809f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" Nov 03 15:16:14.056623 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:14.056586 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vczq8\" (UniqueName: \"kubernetes.io/projected/aaa56c30-5196-48ae-b337-220703d6809f-kube-api-access-vczq8\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t7kz9\" (UID: \"aaa56c30-5196-48ae-b337-220703d6809f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" Nov 03 15:16:14.056623 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:14.056624 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaa56c30-5196-48ae-b337-220703d6809f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t7kz9\" (UID: \"aaa56c30-5196-48ae-b337-220703d6809f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" Nov 03 15:16:14.056825 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:14.056646 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaa56c30-5196-48ae-b337-220703d6809f-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t7kz9\" (UID: \"aaa56c30-5196-48ae-b337-220703d6809f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" Nov 03 15:16:14.056825 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:14.056669 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaa56c30-5196-48ae-b337-220703d6809f-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t7kz9\" (UID: \"aaa56c30-5196-48ae-b337-220703d6809f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" Nov 03 15:16:14.056825 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:14.056801 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaa56c30-5196-48ae-b337-220703d6809f-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t7kz9\" (UID: \"aaa56c30-5196-48ae-b337-220703d6809f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" Nov 03 15:16:14.057024 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:14.056879 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaa56c30-5196-48ae-b337-220703d6809f-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t7kz9\" (UID: \"aaa56c30-5196-48ae-b337-220703d6809f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" Nov 03 15:16:14.057024 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:14.056902 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaa56c30-5196-48ae-b337-220703d6809f-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t7kz9\" (UID: \"aaa56c30-5196-48ae-b337-220703d6809f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" Nov 03 15:16:14.057024 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:14.056934 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaa56c30-5196-48ae-b337-220703d6809f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t7kz9\" (UID: \"aaa56c30-5196-48ae-b337-220703d6809f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" Nov 03 15:16:14.057195 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:14.057037 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaa56c30-5196-48ae-b337-220703d6809f-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t7kz9\" (UID: \"aaa56c30-5196-48ae-b337-220703d6809f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" Nov 03 15:16:14.057195 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:14.057117 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaa56c30-5196-48ae-b337-220703d6809f-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t7kz9\" (UID: \"aaa56c30-5196-48ae-b337-220703d6809f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" Nov 03 15:16:14.057308 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:14.057244 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaa56c30-5196-48ae-b337-220703d6809f-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t7kz9\" (UID: \"aaa56c30-5196-48ae-b337-220703d6809f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" Nov 03 15:16:14.057308 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:14.057296 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaa56c30-5196-48ae-b337-220703d6809f-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t7kz9\" (UID: \"aaa56c30-5196-48ae-b337-220703d6809f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" Nov 03 15:16:14.057524 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:14.057495 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaa56c30-5196-48ae-b337-220703d6809f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t7kz9\" (UID: \"aaa56c30-5196-48ae-b337-220703d6809f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" Nov 03 15:16:14.057646 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:14.057625 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaa56c30-5196-48ae-b337-220703d6809f-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t7kz9\" (UID: \"aaa56c30-5196-48ae-b337-220703d6809f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" Nov 03 15:16:14.059159 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:14.059137 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaa56c30-5196-48ae-b337-220703d6809f-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t7kz9\" (UID: \"aaa56c30-5196-48ae-b337-220703d6809f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" Nov 03 15:16:14.059374 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:14.059355 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaa56c30-5196-48ae-b337-220703d6809f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t7kz9\" (UID: \"aaa56c30-5196-48ae-b337-220703d6809f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" Nov 03 15:16:14.078658 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:14.078633 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaa56c30-5196-48ae-b337-220703d6809f-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t7kz9\" (UID: \"aaa56c30-5196-48ae-b337-220703d6809f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" Nov 03 15:16:14.078873 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:14.078853 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vczq8\" (UniqueName: \"kubernetes.io/projected/aaa56c30-5196-48ae-b337-220703d6809f-kube-api-access-vczq8\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-t7kz9\" (UID: \"aaa56c30-5196-48ae-b337-220703d6809f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" Nov 03 15:16:14.147360 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:14.147281 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" Nov 03 15:16:14.462785 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:14.462759 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9"] Nov 03 15:16:14.464983 ip-10-0-143-171 kubenswrapper[5060]: W1103 15:16:14.464955 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa56c30_5196_48ae_b337_220703d6809f.slice/crio-dd0360dde33f94187d749efe07e9923a6f576c3943953c4051211c6ffe2a2f94 WatchSource:0}: Error finding container dd0360dde33f94187d749efe07e9923a6f576c3943953c4051211c6ffe2a2f94: Status 404 returned error can't find the container with id dd0360dde33f94187d749efe07e9923a6f576c3943953c4051211c6ffe2a2f94 Nov 03 15:16:14.466962 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:14.466932 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:16:14.467038 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:14.466995 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:16:14.467038 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:14.467027 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:16:14.861593 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:14.861559 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" event={"ID":"aaa56c30-5196-48ae-b337-220703d6809f","Type":"ContainerStarted","Data":"b85e64023a1cdeaf83be487aa34f125fb3de38299444b9592b27517bfedd991c"} Nov 03 15:16:14.862069 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:14.861617 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" event={"ID":"aaa56c30-5196-48ae-b337-220703d6809f","Type":"ContainerStarted","Data":"dd0360dde33f94187d749efe07e9923a6f576c3943953c4051211c6ffe2a2f94"} Nov 03 15:16:14.909868 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:14.909780 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" podStartSLOduration=1.9097595250000001 podStartE2EDuration="1.909759525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:16:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:16:14.909156589 +0000 UTC m=+3908.797946171" watchObservedRunningTime="2026-11-03 15:16:14.909759525 +0000 UTC m=+3908.798549107" Nov 03 15:16:15.147764 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:15.147676 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" Nov 03 15:16:15.149314 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:15.149293 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 03 15:16:15.149372 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:15.149339 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" podUID="aaa56c30-5196-48ae-b337-220703d6809f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 03 15:16:16.148349 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:16.148311 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 03 15:16:16.148873 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:16.148369 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" podUID="aaa56c30-5196-48ae-b337-220703d6809f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 03 15:16:17.148616 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:17.148578 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 03 15:16:17.149022 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:17.148637 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" podUID="aaa56c30-5196-48ae-b337-220703d6809f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 03 15:16:18.148253 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:18.148213 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 03 15:16:18.148416 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:18.148279 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" podUID="aaa56c30-5196-48ae-b337-220703d6809f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 03 15:16:19.148471 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:19.148430 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 03 15:16:19.148939 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:19.148490 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" podUID="aaa56c30-5196-48ae-b337-220703d6809f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 03 15:16:20.148484 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:20.148445 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 03 15:16:20.148925 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:20.148511 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" podUID="aaa56c30-5196-48ae-b337-220703d6809f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 03 15:16:21.147694 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:21.147653 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 03 15:16:21.147899 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:21.147730 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" podUID="aaa56c30-5196-48ae-b337-220703d6809f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 03 15:16:22.148427 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:22.148387 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 03 15:16:22.148875 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:22.148442 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" podUID="aaa56c30-5196-48ae-b337-220703d6809f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 03 15:16:23.947786 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:23.947747 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" Nov 03 15:16:23.948301 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:23.948268 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" Nov 03 15:16:23.949020 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:23.948995 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" Nov 03 15:16:28.874718 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:28.874683 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr"] Nov 03 15:16:28.878571 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:28.878547 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" Nov 03 15:16:28.906489 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:28.906457 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr"] Nov 03 15:16:29.009239 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:29.009203 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01ff84be-6243-4ae4-b688-fc4c9402d923-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7g4cr\" (UID: \"01ff84be-6243-4ae4-b688-fc4c9402d923\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" Nov 03 15:16:29.009239 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:29.009245 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01ff84be-6243-4ae4-b688-fc4c9402d923-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7g4cr\" (UID: \"01ff84be-6243-4ae4-b688-fc4c9402d923\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" Nov 03 15:16:29.009437 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:29.009334 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ff84be-6243-4ae4-b688-fc4c9402d923-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7g4cr\" (UID: \"01ff84be-6243-4ae4-b688-fc4c9402d923\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" Nov 03 15:16:29.009437 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:29.009359 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01ff84be-6243-4ae4-b688-fc4c9402d923-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7g4cr\" (UID: \"01ff84be-6243-4ae4-b688-fc4c9402d923\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" Nov 03 15:16:29.009437 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:29.009385 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6wdn\" (UniqueName: \"kubernetes.io/projected/01ff84be-6243-4ae4-b688-fc4c9402d923-kube-api-access-p6wdn\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7g4cr\" (UID: \"01ff84be-6243-4ae4-b688-fc4c9402d923\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" Nov 03 15:16:29.009546 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:29.009478 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01ff84be-6243-4ae4-b688-fc4c9402d923-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7g4cr\" (UID: \"01ff84be-6243-4ae4-b688-fc4c9402d923\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" Nov 03 15:16:29.009546 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:29.009519 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01ff84be-6243-4ae4-b688-fc4c9402d923-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7g4cr\" (UID: \"01ff84be-6243-4ae4-b688-fc4c9402d923\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" Nov 03 15:16:29.009607 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:29.009579 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ff84be-6243-4ae4-b688-fc4c9402d923-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7g4cr\" (UID: \"01ff84be-6243-4ae4-b688-fc4c9402d923\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" Nov 03 15:16:29.009644 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:29.009618 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01ff84be-6243-4ae4-b688-fc4c9402d923-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7g4cr\" (UID: \"01ff84be-6243-4ae4-b688-fc4c9402d923\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" Nov 03 15:16:29.110737 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:29.110709 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ff84be-6243-4ae4-b688-fc4c9402d923-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7g4cr\" (UID: \"01ff84be-6243-4ae4-b688-fc4c9402d923\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" Nov 03 15:16:29.110925 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:29.110741 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01ff84be-6243-4ae4-b688-fc4c9402d923-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7g4cr\" (UID: \"01ff84be-6243-4ae4-b688-fc4c9402d923\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" Nov 03 15:16:29.110925 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:29.110764 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6wdn\" (UniqueName: \"kubernetes.io/projected/01ff84be-6243-4ae4-b688-fc4c9402d923-kube-api-access-p6wdn\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7g4cr\" (UID: \"01ff84be-6243-4ae4-b688-fc4c9402d923\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" Nov 03 15:16:29.110925 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:29.110783 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01ff84be-6243-4ae4-b688-fc4c9402d923-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7g4cr\" (UID: \"01ff84be-6243-4ae4-b688-fc4c9402d923\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" Nov 03 15:16:29.110925 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:29.110910 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01ff84be-6243-4ae4-b688-fc4c9402d923-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7g4cr\" (UID: \"01ff84be-6243-4ae4-b688-fc4c9402d923\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" Nov 03 15:16:29.111434 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:29.110958 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ff84be-6243-4ae4-b688-fc4c9402d923-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7g4cr\" (UID: \"01ff84be-6243-4ae4-b688-fc4c9402d923\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" Nov 03 15:16:29.111434 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:29.111004 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01ff84be-6243-4ae4-b688-fc4c9402d923-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7g4cr\" (UID: \"01ff84be-6243-4ae4-b688-fc4c9402d923\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" Nov 03 15:16:29.111434 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:29.111055 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01ff84be-6243-4ae4-b688-fc4c9402d923-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7g4cr\" (UID: \"01ff84be-6243-4ae4-b688-fc4c9402d923\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" Nov 03 15:16:29.111434 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:29.111091 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01ff84be-6243-4ae4-b688-fc4c9402d923-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7g4cr\" (UID: \"01ff84be-6243-4ae4-b688-fc4c9402d923\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" Nov 03 15:16:29.111434 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:29.111184 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ff84be-6243-4ae4-b688-fc4c9402d923-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7g4cr\" (UID: \"01ff84be-6243-4ae4-b688-fc4c9402d923\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" Nov 03 15:16:29.111434 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:29.111297 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01ff84be-6243-4ae4-b688-fc4c9402d923-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7g4cr\" (UID: \"01ff84be-6243-4ae4-b688-fc4c9402d923\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" Nov 03 15:16:29.111844 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:29.111471 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01ff84be-6243-4ae4-b688-fc4c9402d923-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7g4cr\" (UID: \"01ff84be-6243-4ae4-b688-fc4c9402d923\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" Nov 03 15:16:29.111844 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:29.111522 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ff84be-6243-4ae4-b688-fc4c9402d923-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7g4cr\" (UID: \"01ff84be-6243-4ae4-b688-fc4c9402d923\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" Nov 03 15:16:29.111844 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:29.111587 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01ff84be-6243-4ae4-b688-fc4c9402d923-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7g4cr\" (UID: \"01ff84be-6243-4ae4-b688-fc4c9402d923\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" Nov 03 15:16:29.113203 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:29.113176 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01ff84be-6243-4ae4-b688-fc4c9402d923-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7g4cr\" (UID: \"01ff84be-6243-4ae4-b688-fc4c9402d923\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" Nov 03 15:16:29.113517 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:29.113499 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01ff84be-6243-4ae4-b688-fc4c9402d923-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7g4cr\" (UID: \"01ff84be-6243-4ae4-b688-fc4c9402d923\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" Nov 03 15:16:29.133787 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:29.133715 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6wdn\" (UniqueName: \"kubernetes.io/projected/01ff84be-6243-4ae4-b688-fc4c9402d923-kube-api-access-p6wdn\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7g4cr\" (UID: \"01ff84be-6243-4ae4-b688-fc4c9402d923\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" Nov 03 15:16:29.134304 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:29.134278 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01ff84be-6243-4ae4-b688-fc4c9402d923-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-7g4cr\" (UID: \"01ff84be-6243-4ae4-b688-fc4c9402d923\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" Nov 03 15:16:29.193310 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:29.193274 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" Nov 03 15:16:29.509328 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:29.509301 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr"] Nov 03 15:16:29.512218 ip-10-0-143-171 kubenswrapper[5060]: W1103 15:16:29.512183 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01ff84be_6243_4ae4_b688_fc4c9402d923.slice/crio-7f00e14953a8d9a9657c65accf91c24055976137aea98e19a60b8c9f2fa2011d WatchSource:0}: Error finding container 7f00e14953a8d9a9657c65accf91c24055976137aea98e19a60b8c9f2fa2011d: Status 404 returned error can't find the container with id 7f00e14953a8d9a9657c65accf91c24055976137aea98e19a60b8c9f2fa2011d Nov 03 15:16:29.516650 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:29.516613 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:16:29.516752 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:29.516679 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:16:29.516752 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:29.516707 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:16:29.937662 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:29.937578 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" event={"ID":"01ff84be-6243-4ae4-b688-fc4c9402d923","Type":"ContainerStarted","Data":"46177f6e7b57c4a8d16ed5ac5d515842f3792b42b1e02ead9214289990e0edb0"} Nov 03 15:16:29.937662 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:29.937639 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" event={"ID":"01ff84be-6243-4ae4-b688-fc4c9402d923","Type":"ContainerStarted","Data":"7f00e14953a8d9a9657c65accf91c24055976137aea98e19a60b8c9f2fa2011d"} Nov 03 15:16:29.984231 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:29.984177 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" podStartSLOduration=1.9841622349999999 podStartE2EDuration="1.984162235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:16:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:16:29.982864139 +0000 UTC m=+3923.871653721" watchObservedRunningTime="2026-11-03 15:16:29.984162235 +0000 UTC m=+3923.872951813" Nov 03 15:16:30.193919 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:30.193803 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" Nov 03 15:16:30.195476 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:30.195449 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 03 15:16:30.195595 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:30.195499 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" podUID="01ff84be-6243-4ae4-b688-fc4c9402d923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 03 15:16:31.193736 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:31.193704 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 03 15:16:31.194125 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:31.193768 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" podUID="01ff84be-6243-4ae4-b688-fc4c9402d923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 03 15:16:32.194104 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:32.194066 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 03 15:16:32.194481 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:32.194122 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" podUID="01ff84be-6243-4ae4-b688-fc4c9402d923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 03 15:16:33.193978 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:33.193938 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 03 15:16:33.194184 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:33.193995 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" podUID="01ff84be-6243-4ae4-b688-fc4c9402d923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 03 15:16:34.194503 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:34.194464 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 03 15:16:34.194904 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:34.194518 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" podUID="01ff84be-6243-4ae4-b688-fc4c9402d923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 03 15:16:35.193722 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:35.193681 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 03 15:16:35.193934 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:35.193744 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" podUID="01ff84be-6243-4ae4-b688-fc4c9402d923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 03 15:16:36.194374 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:36.194339 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 03 15:16:36.194785 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:36.194402 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" podUID="01ff84be-6243-4ae4-b688-fc4c9402d923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 03 15:16:37.193711 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:37.193676 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 03 15:16:37.193912 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:37.193727 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" podUID="01ff84be-6243-4ae4-b688-fc4c9402d923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 03 15:16:39.056604 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:39.056562 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" Nov 03 15:16:39.057327 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:39.057303 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" Nov 03 15:16:39.058129 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:16:39.058110 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" Nov 03 15:16:39.438631 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:16:39.438545 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:17:09.439086 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:17:09.439046 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:17:39.481813 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:17:39.481778 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:17:39.988991 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:17:39.988954 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:17:39.989170 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:17:39.989009 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" podUID="aaa56c30-5196-48ae-b337-220703d6809f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:17:40.104055 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:17:40.104006 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:17:40.104250 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:17:40.104075 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" podUID="01ff84be-6243-4ae4-b688-fc4c9402d923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:17:59.146338 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:17:59.146262 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb"] Nov 03 15:17:59.150051 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:17:59.150030 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" Nov 03 15:17:59.178406 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:17:59.178378 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb"] Nov 03 15:17:59.243692 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:17:59.243662 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36997fee-48b5-4f04-9b26-98b4c45fadea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-cthxb\" (UID: \"36997fee-48b5-4f04-9b26-98b4c45fadea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" Nov 03 15:17:59.243870 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:17:59.243698 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgkvr\" (UniqueName: \"kubernetes.io/projected/36997fee-48b5-4f04-9b26-98b4c45fadea-kube-api-access-vgkvr\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-cthxb\" (UID: \"36997fee-48b5-4f04-9b26-98b4c45fadea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" Nov 03 15:17:59.243870 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:17:59.243723 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36997fee-48b5-4f04-9b26-98b4c45fadea-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-cthxb\" (UID: \"36997fee-48b5-4f04-9b26-98b4c45fadea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" Nov 03 15:17:59.243870 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:17:59.243827 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36997fee-48b5-4f04-9b26-98b4c45fadea-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-cthxb\" (UID: \"36997fee-48b5-4f04-9b26-98b4c45fadea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" Nov 03 15:17:59.244039 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:17:59.243879 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36997fee-48b5-4f04-9b26-98b4c45fadea-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-cthxb\" (UID: \"36997fee-48b5-4f04-9b26-98b4c45fadea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" Nov 03 15:17:59.244039 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:17:59.243904 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36997fee-48b5-4f04-9b26-98b4c45fadea-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-cthxb\" (UID: \"36997fee-48b5-4f04-9b26-98b4c45fadea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" Nov 03 15:17:59.244039 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:17:59.243926 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36997fee-48b5-4f04-9b26-98b4c45fadea-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-cthxb\" (UID: \"36997fee-48b5-4f04-9b26-98b4c45fadea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" Nov 03 15:17:59.244039 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:17:59.243971 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36997fee-48b5-4f04-9b26-98b4c45fadea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-cthxb\" (UID: \"36997fee-48b5-4f04-9b26-98b4c45fadea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" Nov 03 15:17:59.244217 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:17:59.244075 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36997fee-48b5-4f04-9b26-98b4c45fadea-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-cthxb\" (UID: \"36997fee-48b5-4f04-9b26-98b4c45fadea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" Nov 03 15:17:59.344644 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:17:59.344607 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36997fee-48b5-4f04-9b26-98b4c45fadea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-cthxb\" (UID: \"36997fee-48b5-4f04-9b26-98b4c45fadea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" Nov 03 15:17:59.344831 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:17:59.344651 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgkvr\" (UniqueName: \"kubernetes.io/projected/36997fee-48b5-4f04-9b26-98b4c45fadea-kube-api-access-vgkvr\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-cthxb\" (UID: \"36997fee-48b5-4f04-9b26-98b4c45fadea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" Nov 03 15:17:59.344831 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:17:59.344688 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36997fee-48b5-4f04-9b26-98b4c45fadea-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-cthxb\" (UID: \"36997fee-48b5-4f04-9b26-98b4c45fadea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" Nov 03 15:17:59.344831 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:17:59.344739 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36997fee-48b5-4f04-9b26-98b4c45fadea-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-cthxb\" (UID: \"36997fee-48b5-4f04-9b26-98b4c45fadea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" Nov 03 15:17:59.344831 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:17:59.344770 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36997fee-48b5-4f04-9b26-98b4c45fadea-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-cthxb\" (UID: \"36997fee-48b5-4f04-9b26-98b4c45fadea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" Nov 03 15:17:59.344831 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:17:59.344805 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36997fee-48b5-4f04-9b26-98b4c45fadea-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-cthxb\" (UID: \"36997fee-48b5-4f04-9b26-98b4c45fadea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" Nov 03 15:17:59.344831 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:17:59.344864 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36997fee-48b5-4f04-9b26-98b4c45fadea-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-cthxb\" (UID: \"36997fee-48b5-4f04-9b26-98b4c45fadea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" Nov 03 15:17:59.345205 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:17:59.344888 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36997fee-48b5-4f04-9b26-98b4c45fadea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-cthxb\" (UID: \"36997fee-48b5-4f04-9b26-98b4c45fadea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" Nov 03 15:17:59.345259 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:17:59.345201 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36997fee-48b5-4f04-9b26-98b4c45fadea-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-cthxb\" (UID: \"36997fee-48b5-4f04-9b26-98b4c45fadea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" Nov 03 15:17:59.345259 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:17:59.345229 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36997fee-48b5-4f04-9b26-98b4c45fadea-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-cthxb\" (UID: \"36997fee-48b5-4f04-9b26-98b4c45fadea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" Nov 03 15:17:59.345354 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:17:59.345241 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36997fee-48b5-4f04-9b26-98b4c45fadea-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-cthxb\" (UID: \"36997fee-48b5-4f04-9b26-98b4c45fadea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" Nov 03 15:17:59.345354 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:17:59.345336 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36997fee-48b5-4f04-9b26-98b4c45fadea-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-cthxb\" (UID: \"36997fee-48b5-4f04-9b26-98b4c45fadea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" Nov 03 15:17:59.345460 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:17:59.345404 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36997fee-48b5-4f04-9b26-98b4c45fadea-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-cthxb\" (UID: \"36997fee-48b5-4f04-9b26-98b4c45fadea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" Nov 03 15:17:59.345732 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:17:59.345711 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36997fee-48b5-4f04-9b26-98b4c45fadea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-cthxb\" (UID: \"36997fee-48b5-4f04-9b26-98b4c45fadea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" Nov 03 15:17:59.347255 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:17:59.347228 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36997fee-48b5-4f04-9b26-98b4c45fadea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-cthxb\" (UID: \"36997fee-48b5-4f04-9b26-98b4c45fadea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" Nov 03 15:17:59.347255 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:17:59.347251 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36997fee-48b5-4f04-9b26-98b4c45fadea-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-cthxb\" (UID: \"36997fee-48b5-4f04-9b26-98b4c45fadea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" Nov 03 15:17:59.367287 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:17:59.367263 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36997fee-48b5-4f04-9b26-98b4c45fadea-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-cthxb\" (UID: \"36997fee-48b5-4f04-9b26-98b4c45fadea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" Nov 03 15:17:59.367516 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:17:59.367492 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgkvr\" (UniqueName: \"kubernetes.io/projected/36997fee-48b5-4f04-9b26-98b4c45fadea-kube-api-access-vgkvr\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-cthxb\" (UID: \"36997fee-48b5-4f04-9b26-98b4c45fadea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" Nov 03 15:17:59.462742 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:17:59.462670 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" Nov 03 15:17:59.782375 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:17:59.782233 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb"] Nov 03 15:17:59.785159 ip-10-0-143-171 kubenswrapper[5060]: W1103 15:17:59.785131 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36997fee_48b5_4f04_9b26_98b4c45fadea.slice/crio-89c16eba63a2512bd7598b56fae459f302b60d1e89fca669d33f9fdc7ec1f852 WatchSource:0}: Error finding container 89c16eba63a2512bd7598b56fae459f302b60d1e89fca669d33f9fdc7ec1f852: Status 404 returned error can't find the container with id 89c16eba63a2512bd7598b56fae459f302b60d1e89fca669d33f9fdc7ec1f852 Nov 03 15:17:59.787254 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:17:59.787225 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:17:59.787316 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:17:59.787291 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:17:59.787351 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:17:59.787320 5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 15:18:00.425214 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:18:00.425176 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" event={"ID":"36997fee-48b5-4f04-9b26-98b4c45fadea","Type":"ContainerStarted","Data":"fea29f5512a469016351e007b6f91f545cd1f8de7c1c870998a517804dd48166"} Nov 03 15:18:00.425738 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:18:00.425234 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" event={"ID":"36997fee-48b5-4f04-9b26-98b4c45fadea","Type":"ContainerStarted","Data":"89c16eba63a2512bd7598b56fae459f302b60d1e89fca669d33f9fdc7ec1f852"} Nov 03 15:18:00.463085 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:18:00.463055 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" Nov 03 15:18:00.464581 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:18:00.464558 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-cthxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 15:18:00.464705 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:18:00.464603 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" podUID="36997fee-48b5-4f04-9b26-98b4c45fadea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 15:18:00.522881 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:18:00.510605 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" podStartSLOduration=1.510585028 podStartE2EDuration="1.510585028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:17:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:18:00.484674574 +0000 UTC m=+4014.373464155" watchObservedRunningTime="2026-11-03 15:18:00.510585028 +0000 UTC m=+4014.399374611" Nov 03 15:18:01.463174 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:18:01.463138 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-cthxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 15:18:01.463656 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:18:01.463193 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" podUID="36997fee-48b5-4f04-9b26-98b4c45fadea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 15:18:02.463851 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:18:02.463798 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-cthxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 15:18:02.464244 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:18:02.463890 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" podUID="36997fee-48b5-4f04-9b26-98b4c45fadea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 15:18:03.463896 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:18:03.463862 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-cthxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 15:18:03.464314 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:18:03.463927 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" podUID="36997fee-48b5-4f04-9b26-98b4c45fadea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 15:18:04.463209 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:18:04.463171 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-cthxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 15:18:04.463416 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:18:04.463236 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" podUID="36997fee-48b5-4f04-9b26-98b4c45fadea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 15:18:05.463275 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:18:05.463239 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-cthxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 15:18:05.463696 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:18:05.463287 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" podUID="36997fee-48b5-4f04-9b26-98b4c45fadea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 15:18:06.463227 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:18:06.463193 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-cthxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 15:18:06.463398 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:18:06.463252 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" podUID="36997fee-48b5-4f04-9b26-98b4c45fadea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 15:18:07.463418 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:18:07.463380 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-cthxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 15:18:07.463932 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:18:07.463436 5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" podUID="36997fee-48b5-4f04-9b26-98b4c45fadea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 15:18:09.004643 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:18:09.004606 5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" Nov 03 15:18:09.005146 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:18:09.005126 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" Nov 03 15:18:09.005978 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:18:09.005953 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" Nov 03 15:18:09.439730 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:18:09.439641 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:18:39.438865 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:18:39.438811 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:19:09.440808 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:19:09.440774 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:19:39.438122 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:19:39.438083 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:20:09.439426 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:20:09.439389 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:20:39.439809 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:20:39.439763 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:21:07.718890 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:21:07.718754 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/3.log" Nov 03 15:21:07.728078 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:21:07.728059 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/3.log" Nov 03 15:21:09.439487 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:21:09.439440 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:21:39.437768 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:21:39.437729 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:21:55.048025 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:21:55.047987 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-cthxb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:21:55.048748 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:21:55.048095 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" podUID="36997fee-48b5-4f04-9b26-98b4c45fadea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:22:09.443111 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:22:09.443072 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:22:39.438961 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:22:39.438921 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:23:09.496879 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:23:09.496818 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:23:09.996007 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:23:09.995970 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:23:09.996178 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:23:09.996042 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-t7kz9" podUID="aaa56c30-5196-48ae-b337-220703d6809f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:23:10.052081 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:23:10.051939 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-cthxb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:23:10.052081 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:23:10.052010 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" podUID="36997fee-48b5-4f04-9b26-98b4c45fadea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:23:10.107988 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:23:10.107936 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:23:10.108340 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:23:10.108305 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" podUID="01ff84be-6243-4ae4-b688-fc4c9402d923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:23:39.438281 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:23:39.438247 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:24:09.440192 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:24:09.440121 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:24:39.487007 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:24:39.486970 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:24:40.053866 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:24:40.052940 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-cthxb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:24:40.053866 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:24:40.053053 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-cthxb" podUID="36997fee-48b5-4f04-9b26-98b4c45fadea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:24:40.118859 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:24:40.117946 5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:24:40.118859 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:24:40.118025 5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-7g4cr" podUID="01ff84be-6243-4ae4-b688-fc4c9402d923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:25:09.445816 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:25:09.445774 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:25:39.438983 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:25:39.438947 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:26:07.751948 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:26:07.751922 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/3.log" Nov 03 15:26:07.763013 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:26:07.762993 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/3.log" Nov 03 15:26:09.438869 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:26:09.438818 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:26:33.343944 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:26:33.343911 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-4ggmd_7b8c342f-20c2-406f-a6a1-f8eef4118a6d/manager/0.log" Nov 03 15:26:33.777581 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:26:33.777555 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-54d5w_218bc458-4d44-453a-84c8-2b7fe211580e/manager/0.log" Nov 03 15:26:35.627729 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:26:35.627692 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-84b5dd6748-lx27t_5a1d1b5e-f53f-40fb-9ec9-a3d40d0e8549/maas-api/0.log" Nov 03 15:26:39.439103 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:26:39.439055 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:26:41.383739 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:26:41.383710 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-jf2qh_37e492af-6e43-4224-be07-6e92f69bbae1/manager/0.log" Nov 03 15:26:41.596342 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:26:41.596313 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-t7s86_57e263ee-fe23-475e-aa0d-72f40f837809/manager/0.log" Nov 03 15:26:42.004546 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:26:42.004500 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-5rnng_af191281-ee95-447a-9713-4a922592fe53/manager/0.log" Nov 03 15:26:42.435910 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:26:42.435824 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-569f5_3ced846c-60f0-4206-836e-f94c281d6438/manager/0.log" Nov 03 15:26:44.387744 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:26:44.387708 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-7g4cr_01ff84be-6243-4ae4-b688-fc4c9402d923/istio-proxy/0.log" Nov 03 15:26:44.994931 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:26:44.994889 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-cthxb_36997fee-48b5-4f04-9b26-98b4c45fadea/istio-proxy/0.log" Nov 03 15:26:45.578985 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:26:45.578947 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-t7kz9_aaa56c30-5196-48ae-b337-220703d6809f/istio-proxy/0.log" Nov 03 15:26:46.697024 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:26:46.696995 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6c7f54c9b7-t6wbf_36065516-87ab-4057-b198-439305606d12/router/0.log" Nov 03 15:27:03.610053 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:03.609978 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2xrr4_ef43449c-b7dc-4e84-b785-63201f80d4da/global-pull-secret-syncer/0.log" Nov 03 15:27:04.172217 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:04.172186 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-g28dw_00a412a1-3764-404b-9e6d-3a2bbb200781/connection-checker/0.log" Nov 03 15:27:04.305979 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:04.305949 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-pkhg9_47c859b2-a25b-45e5-acdc-0632d472c097/connection-checker/0.log" Nov 03 15:27:04.446219 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:04.446150 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-wcb9f_4d66aab2-4671-4bbd-a4cc-b9dd9c5a766d/connection-checker/0.log" Nov 03 15:27:04.764481 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:04.764448 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dj8cg_e9dee2ef-6437-4963-8e27-da5fb9c75928/konnectivity-agent/0.log" Nov 03 15:27:05.410321 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:05.410288 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-171.ec2.internal_3a1b8625ac3f3be5fcd1bbb72178f5c6/haproxy/0.log" Nov 03 15:27:07.862088 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:07.862042 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-4ggmd_7b8c342f-20c2-406f-a6a1-f8eef4118a6d/manager/0.log" Nov 03 15:27:08.588052 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:08.588016 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-54d5w_218bc458-4d44-453a-84c8-2b7fe211580e/manager/0.log" Nov 03 15:27:09.438963 ip-10-0-143-171 kubenswrapper[5060]: E1103 15:27:09.438925 5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sm2jg" containerName="extractor" Nov 03 15:27:11.125300 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:11.125268 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-4ggmd_7b8c342f-20c2-406f-a6a1-f8eef4118a6d/manager/0.log" Nov 03 15:27:11.457464 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:11.457375 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-jf2qh_37e492af-6e43-4224-be07-6e92f69bbae1/manager/0.log" Nov 03 15:27:11.601049 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:11.601012 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-t7s86_57e263ee-fe23-475e-aa0d-72f40f837809/manager/0.log" Nov 03 15:27:11.845015 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:11.844987 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-54d5w_218bc458-4d44-453a-84c8-2b7fe211580e/manager/0.log" Nov 03 15:27:12.067342 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:12.067297 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-5rnng_af191281-ee95-447a-9713-4a922592fe53/manager/0.log" Nov 03 15:27:12.514825 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:12.514783 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-569f5_3ced846c-60f0-4206-836e-f94c281d6438/manager/0.log" Nov 03 15:27:13.820943 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:13.820908 5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rfl4d/perf-node-gather-daemonset-rzgnp"] Nov 03 15:27:13.824573 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:13.824551 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-rzgnp" Nov 03 15:27:13.845255 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:13.845217 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rfl4d\"/\"kube-root-ca.crt\"" Nov 03 15:27:13.854645 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:13.854615 5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rfl4d\"/\"openshift-service-ca.crt\"" Nov 03 15:27:13.857182 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:13.857149 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rfl4d/perf-node-gather-daemonset-rzgnp"] Nov 03 15:27:13.914969 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:13.914940 5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rfl4d\"/\"default-dockercfg-nh78f\"" Nov 03 15:27:13.931301 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:13.931275 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrv2l\" (UniqueName: \"kubernetes.io/projected/a882120d-937f-451d-a7a5-988556ce0d9b-kube-api-access-vrv2l\") pod \"perf-node-gather-daemonset-rzgnp\" (UID: \"a882120d-937f-451d-a7a5-988556ce0d9b\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-rzgnp" Nov 03 15:27:13.931464 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:13.931333 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a882120d-937f-451d-a7a5-988556ce0d9b-podres\") pod \"perf-node-gather-daemonset-rzgnp\" (UID: \"a882120d-937f-451d-a7a5-988556ce0d9b\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-rzgnp" Nov 03 15:27:13.931521 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:13.931467 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a882120d-937f-451d-a7a5-988556ce0d9b-sys\") pod \"perf-node-gather-daemonset-rzgnp\" (UID: \"a882120d-937f-451d-a7a5-988556ce0d9b\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-rzgnp" Nov 03 15:27:13.931558 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:13.931514 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a882120d-937f-451d-a7a5-988556ce0d9b-lib-modules\") pod \"perf-node-gather-daemonset-rzgnp\" (UID: \"a882120d-937f-451d-a7a5-988556ce0d9b\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-rzgnp" Nov 03 15:27:13.931607 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:13.931583 5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a882120d-937f-451d-a7a5-988556ce0d9b-proc\") pod \"perf-node-gather-daemonset-rzgnp\" (UID: \"a882120d-937f-451d-a7a5-988556ce0d9b\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-rzgnp" Nov 03 15:27:14.032585 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:14.032548 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a882120d-937f-451d-a7a5-988556ce0d9b-podres\") pod \"perf-node-gather-daemonset-rzgnp\" (UID: \"a882120d-937f-451d-a7a5-988556ce0d9b\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-rzgnp" Nov 03 15:27:14.032764 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:14.032683 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a882120d-937f-451d-a7a5-988556ce0d9b-podres\") pod \"perf-node-gather-daemonset-rzgnp\" (UID: \"a882120d-937f-451d-a7a5-988556ce0d9b\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-rzgnp" Nov 03 15:27:14.032764 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:14.032703 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a882120d-937f-451d-a7a5-988556ce0d9b-sys\") pod \"perf-node-gather-daemonset-rzgnp\" (UID: \"a882120d-937f-451d-a7a5-988556ce0d9b\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-rzgnp" Nov 03 15:27:14.032764 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:14.032738 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a882120d-937f-451d-a7a5-988556ce0d9b-sys\") pod \"perf-node-gather-daemonset-rzgnp\" (UID: \"a882120d-937f-451d-a7a5-988556ce0d9b\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-rzgnp" Nov 03 15:27:14.032764 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:14.032747 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a882120d-937f-451d-a7a5-988556ce0d9b-lib-modules\") pod \"perf-node-gather-daemonset-rzgnp\" (UID: \"a882120d-937f-451d-a7a5-988556ce0d9b\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-rzgnp" Nov 03 15:27:14.032937 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:14.032774 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a882120d-937f-451d-a7a5-988556ce0d9b-proc\") pod \"perf-node-gather-daemonset-rzgnp\" (UID: \"a882120d-937f-451d-a7a5-988556ce0d9b\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-rzgnp" Nov 03 15:27:14.032937 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:14.032798 5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrv2l\" (UniqueName: \"kubernetes.io/projected/a882120d-937f-451d-a7a5-988556ce0d9b-kube-api-access-vrv2l\") pod \"perf-node-gather-daemonset-rzgnp\" (UID: \"a882120d-937f-451d-a7a5-988556ce0d9b\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-rzgnp" Nov 03 15:27:14.032937 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:14.032848 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a882120d-937f-451d-a7a5-988556ce0d9b-lib-modules\") pod \"perf-node-gather-daemonset-rzgnp\" (UID: \"a882120d-937f-451d-a7a5-988556ce0d9b\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-rzgnp" Nov 03 15:27:14.032937 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:14.032886 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a882120d-937f-451d-a7a5-988556ce0d9b-proc\") pod \"perf-node-gather-daemonset-rzgnp\" (UID: \"a882120d-937f-451d-a7a5-988556ce0d9b\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-rzgnp" Nov 03 15:27:14.057283 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:14.057249 5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrv2l\" (UniqueName: \"kubernetes.io/projected/a882120d-937f-451d-a7a5-988556ce0d9b-kube-api-access-vrv2l\") pod \"perf-node-gather-daemonset-rzgnp\" (UID: \"a882120d-937f-451d-a7a5-988556ce0d9b\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-rzgnp" Nov 03 15:27:14.137240 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:14.137155 5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-rzgnp" Nov 03 15:27:14.451166 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:14.451140 5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rfl4d/perf-node-gather-daemonset-rzgnp"] Nov 03 15:27:14.453664 ip-10-0-143-171 kubenswrapper[5060]: W1103 15:27:14.453632 5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda882120d_937f_451d_a7a5_988556ce0d9b.slice/crio-e93175e7d1ef19306810a6dfe03e0155ec4ca46a4d8227c10c67914fe8eb698b WatchSource:0}: Error finding container e93175e7d1ef19306810a6dfe03e0155ec4ca46a4d8227c10c67914fe8eb698b: Status 404 returned error can't find the container with id e93175e7d1ef19306810a6dfe03e0155ec4ca46a4d8227c10c67914fe8eb698b Nov 03 15:27:14.607327 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:14.607299 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-4ggmd_7b8c342f-20c2-406f-a6a1-f8eef4118a6d/manager/0.log" Nov 03 15:27:15.000108 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:15.000062 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-rzgnp" event={"ID":"a882120d-937f-451d-a7a5-988556ce0d9b","Type":"ContainerStarted","Data":"ae7083355614c1e7cf0d98a358096623e3fdddd2898c02d50f646b153369aa45"} Nov 03 15:27:15.000594 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:15.000123 5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-rzgnp" event={"ID":"a882120d-937f-451d-a7a5-988556ce0d9b","Type":"ContainerStarted","Data":"e93175e7d1ef19306810a6dfe03e0155ec4ca46a4d8227c10c67914fe8eb698b"} Nov 03 15:27:15.000594 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:15.000171 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-rzgnp" Nov 03 15:27:15.047593 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:15.047538 5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-rzgnp" podStartSLOduration=2.047525918 podStartE2EDuration="2.047525918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 15:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:27:15.04542293 +0000 UTC m=+4568.934212514" watchObservedRunningTime="2026-11-03 15:27:15.047525918 +0000 UTC m=+4568.936315491" Nov 03 15:27:15.178897 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:15.178859 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-v6pjk_bdf183c4-a9ee-4ea8-ace1-50f307114328/cluster-monitoring-operator/0.log" Nov 03 15:27:15.312488 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:15.312404 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-54d5w_218bc458-4d44-453a-84c8-2b7fe211580e/manager/0.log" Nov 03 15:27:16.950239 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:16.950212 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-znvl4_95d09114-670c-4ca3-bbcc-f41b5f8464e1/node-exporter/0.log" Nov 03 15:27:17.095288 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:17.095256 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-znvl4_95d09114-670c-4ca3-bbcc-f41b5f8464e1/kube-rbac-proxy/0.log" Nov 03 15:27:17.238708 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:17.238637 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-znvl4_95d09114-670c-4ca3-bbcc-f41b5f8464e1/init-textfile/0.log" Nov 03 15:27:18.900598 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:18.900568 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-dh22t_aea82995-1556-4f76-a701-53146ad84381/prometheus-operator/0.log" Nov 03 15:27:19.043243 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:19.043211 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-dh22t_aea82995-1556-4f76-a701-53146ad84381/kube-rbac-proxy/0.log" Nov 03 15:27:19.195386 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:19.195312 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-9bb7m_f5699d04-232e-4267-9c1b-8b206a22fa11/prometheus-operator-admission-webhook/0.log" Nov 03 15:27:21.017449 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:21.017422 5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-rzgnp" Nov 03 15:27:21.401394 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:21.401315 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-7g4cr_01ff84be-6243-4ae4-b688-fc4c9402d923/istio-proxy/0.log" Nov 03 15:27:21.629735 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:21.629692 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-cthxb_36997fee-48b5-4f04-9b26-98b4c45fadea/istio-proxy/0.log" Nov 03 15:27:21.865752 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:21.865721 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-t7kz9_aaa56c30-5196-48ae-b337-220703d6809f/istio-proxy/0.log" Nov 03 15:27:22.288466 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:22.288434 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-zgd4k_6f219713-6fd8-4580-b1c0-24e2a929c258/networking-console-plugin/0.log" Nov 03 15:27:22.322585 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:22.322556 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6c7f54c9b7-t6wbf_36065516-87ab-4057-b198-439305606d12/router/0.log" Nov 03 15:27:23.630481 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:23.630447 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/3.log" Nov 03 15:27:23.637713 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:23.637685 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5xzwg_307e48f6-4a66-4c9c-a4ff-848cce9d3e0e/console-operator/4.log" Nov 03 15:27:25.160933 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:25.160898 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-jf2qh_37e492af-6e43-4224-be07-6e92f69bbae1/manager/0.log" Nov 03 15:27:25.240723 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:25.240689 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-t7s86_57e263ee-fe23-475e-aa0d-72f40f837809/manager/0.log" Nov 03 15:27:25.502249 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:25.502217 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-5rnng_af191281-ee95-447a-9713-4a922592fe53/manager/0.log" Nov 03 15:27:25.765358 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:25.765280 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-569f5_3ced846c-60f0-4206-836e-f94c281d6438/manager/0.log" Nov 03 15:27:26.377973 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:26.377940 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-v9mxv_11d16990-ce0c-458f-ab1e-e9a3fed3221b/volume-data-source-validator/0.log" Nov 03 15:27:29.074969 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:29.074939 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2bk97_61060804-c519-490a-ac90-674792d50174/dns/0.log" Nov 03 15:27:29.212937 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:29.212908 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2bk97_61060804-c519-490a-ac90-674792d50174/kube-rbac-proxy/0.log" Nov 03 15:27:30.099338 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:30.099301 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q4gsb_e637e650-5fc6-4adb-a97a-47d924c30f97/dns-node-resolver/0.log" Nov 03 15:27:31.300051 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:31.300024 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-7g4cr_01ff84be-6243-4ae4-b688-fc4c9402d923/istio-proxy/0.log" Nov 03 15:27:31.529381 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:31.529351 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-cthxb_36997fee-48b5-4f04-9b26-98b4c45fadea/istio-proxy/0.log" Nov 03 15:27:31.639082 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:31.639000 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7c486d9667-j4szh_cd3973a6-477c-4cbf-88eb-1cf438e2bee6/registry/0.log" Nov 03 15:27:31.781855 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:31.781811 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-t7kz9_aaa56c30-5196-48ae-b337-220703d6809f/istio-proxy/0.log" Nov 03 15:27:32.044974 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:32.044940 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t4b94_e68be0c4-d4ee-4da5-82e3-82372f811841/node-ca/0.log" Nov 03 15:27:32.232216 ip-10-0-143-171 kubenswrapper[5060]: I1103 15:27:32.232183 5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6c7f54c9b7-t6wbf_36065516-87ab-4057-b198-439305606d12/router/0.log"